Latest / Agentic Conversations (formally mlops.community) / A Candid Conversation Around MCP and A2A // Rahul Parundekar and Sam Partee // #316 SF Live

Transcript

A Candid Conversation Around MCP and A2A // Rahul Parundekar and Sam Partee // #316 SF Live · Agentic Conversations (formally mlops.community)

• 0:02
  Business stuff on the From built from the ground up.
• 0:06
  We're editing. Yeah.
• 0:07
  No, no, no. It's all going injured.
• 0:15
  All right, we're not editing now.
• 0:20
  I swear to God like I would love if.
• 0:23
  The chat was just like this instead of us being like.
• 0:26
  Well, we got this kind of little, yeah.
• 0:28
  Of course. No, really.
• 0:29
  Afterwards, right after it's. Like that, the the Bluebird
• 0:32
  reels, the Bluebird reels. Well, we got to sound a little
• 0:35
  right. Yeah.
• 0:36
  Exactly. No, that's what we're leading.
• 0:37
  I have to be. That is the intro right there.
• 0:43
  I've been traveling for weeks and I got to San Francisco.
• 0:48
  I'm here with these two guys. Flight trains and automobiles.
• 0:51
  The thing is, I'm still a little jet lagged so I may doze off
• 0:55
  while we are chatting. Or what?
• 0:57
  Is that boring? If do not take it personally, if
• 1:01
  I do, feel free to wake me up or tap me.
• 1:04
  But this is going to be a conversation all around.
• 1:06
  Tools, the good, the bad and the ugly.
• 1:09
  What's the tool? What is a tool?
• 1:11
  We'll talk about that. We'll talk about agents.
• 1:13
  We'll talk about the new hype waves.
• 1:15
  Love it. I always start with what's an
• 1:17
  agent and like what my definition of that is, because
• 1:19
  if you don't know what like how to define what an agent is, what
• 1:26
  a tool is, is secondary. And so like very simply agent,
• 1:31
  some piece of text, large language bottle, a ability to
• 1:35
  take that text and feed it to a deterministic process, which is
• 1:40
  commonly code and a way to run that.
• 1:43
  So we call that a tool. That function I'm talking about,
• 1:47
  F of X = y, any representative function can be a tool as long
• 1:54
  as the output of a large language model is capable of
• 1:58
  being the input. And then right now the execution
• 2:03
  of that tool, it's really left up to the developer.
• 2:08
  And that's actually one of the problems that stuff like MCP,
• 2:12
  any of the arcade are trying to solve.
• 2:15
  And there's a lot of attempts at this, but beforehand they were
• 2:19
  just in Lang chain and Llama index.
• 2:22
  You just ran it wherever the LLM client called, and that was not
• 2:27
  optimal. I, I really like this
• 2:30
  definition. I think with the tools, the
• 2:33
  agent can now access information that it doesn't have memorized,
• 2:38
  right? And so it's coming in, it's able
• 2:40
  to access stuff on the fly. So which is still very different
• 2:44
  than reasoning. Reasoning is think step by step,
• 2:46
  but tools is how do you retrieve it and then be able to act on
• 2:50
  it. And that suddenly expands the
• 2:54
  scope of how LLMS work, right? So if you've used the latest
• 2:58
  ChatGPT that has tool use built in, because you can see like it
• 3:03
  is searching the web, it is doing stuff, right?
• 3:06
  And imagine the possibilities if it can now access, let's say,
• 3:10
  your private database in the enterprise.
• 3:12
  You have files located somewhere, it can read those.
• 3:15
  All of that is the power giving the the the ability of an agent
• 3:20
  to access information outside of it is what tools enables it.
• 3:24
  Yeah, so now where are tools struggling these days?
• 3:29
  Well. I mean, there's a very good
• 3:31
  point that he's raising, which is he's talking essentially
• 3:33
  about context retrieval and model context protocol.
• 3:37
  And this there's, there's, there's an interesting point in
• 3:43
  this, which is like everybody because of rag, right?
• 3:46
  You think about go get, go get me text to feed into this
• 3:51
  prompt, basically go augment my process.
• 3:54
  But there's some in which you're saying you're talking about, and
• 3:58
  then you take that and you can act.
• 4:01
  And the problems come in when we try to start to use these tools
• 4:05
  to act for people, because MCP is not prepared to act as
• 4:13
  anyone. No servers are prepared to act
• 4:16
  as people, which is why you see them working on authorization
• 4:21
  protocols, why you see them working on Https://streamable
• 4:24
  instead of desktop local. We have to elevate the
• 4:29
  privileges of these agents in order to have that second part
• 4:34
  of tools responsibility, which is like taking action, rather
• 4:37
  than just get in context. Yeah.
• 4:39
  And that's the hardest part amongst all of this, because
• 4:43
  what happens is LMS are not perfect, they're not
• 4:46
  deterministic. And you might expect that when
• 4:49
  you say, well, change this, get change this file in my code for
• 4:52
  me, it might reason and come up with the conclusion that it's
• 4:56
  better to just delete everything.
• 4:57
  Yeah. RF-RF.
• 4:59
  Right. And so you really don't want it
• 5:01
  to do that, right, with your prod database, Like what is it
• 5:03
  going to do? Drop table and then migrate it?
• 5:06
  No, you still want it, right? And so like with humans, with
• 5:10
  humans, you're, you're giving them these privileges to, to, to
• 5:17
  work on things in a very purposeful manner.
• 5:20
  What are we doing with agents? Are you giving them the keys to
• 5:25
  everything and this kitchen sink and like letting it go haywire?
• 5:29
  Obviously not. Yeah, I'm getting anxious when
• 5:31
  you're just saying. Exactly.
• 5:34
  You don't really want it to do that.
• 5:35
  And so that's where like where these tools are currently
• 5:38
  struggling. Unbelievably great point the I
• 5:42
  think every MCP server today, yeah.
• 5:44
  What do they do with What are the tokens smitherey?
• 5:46
  Like what do they make you do on the configuration page?
• 5:49
  Oh, go copy paste your Long live token into this website that
• 5:54
  just appeared a few months ago. That sounds like a great
• 5:57
  practice. You know, you can refresh that
• 6:00
  and like you can access their data for eternity.
• 6:04
  Like that's your Google Drive token you're going to give to
• 6:06
  that. OK, sure.
• 6:08
  And and on top of that, your point is really good.
• 6:10
  Think about an EA as the example I give.
• 6:13
  Like you don't do that to your EA.
• 6:16
  You have delegated privileges, right?
• 6:18
  You don't let them. Like you have a different set
• 6:23
  and just a bot token with a different set doesn't solve it
• 6:26
  either because you don't want the bot's data.
• 6:29
  You want your data. You don't want it to send an
• 6:32
  e-mail as the bot, you want it to send an e-mail as you.
• 6:35
  And so these delegated privileges you're talking about
• 6:38
  is a great point. It's it's what we call tool
• 6:41
  authorization. It's a separate thing than like
• 6:45
  the authorization of accessing a website.
• 6:48
  And it's. It's a really complicated
• 6:50
  problem. But why is it different than O
• 6:53
  auth? It's it's essentially still O
• 6:56
  auth. It's just you have another
• 6:57
  intermediary in the flow, even agent.
• 7:00
  It's not just user site service, it's user agent site service,
• 7:07
  right, right. And there's an intermediary
• 7:09
  there that has to have now a responsibility in this flow.
• 7:14
  And you want to delegate different permissions to the
• 7:16
  agent, correct? As opposed to if I'm going to
• 7:19
  that site and doing Oauth then I have certain.
• 7:24
  Privileges. It used to be basically just log
• 7:27
  into site you get set of privileges.
• 7:30
  It's now you log into site and you can give an agent all types
• 7:37
  of rivileges. There's another layer that has
• 7:42
  its own responsibilities and needs its own scopes and claims
• 7:46
  and ermissions, or else you are stuck with either two.
• 7:51
  You're stuck in two scenarios. Either you give it the whole
• 7:54
  world and you give it all your access, in which case Rahul's
• 7:58
  point comes up. Or if Johnny drop tables or you
• 8:02
  can't do anything, you end up with a bot token.
• 8:05
  And then your tools aren't any good because they're not
• 8:08
  effectively getting the stuff you or doing the stuff you want
• 8:11
  them to do. And, and I'd like to think about
• 8:13
  like who's using it from even the bot perspective.
• 8:18
  For example, you don't want to have only one set of permissions
• 8:21
  that you give to all your bots. You want your cursor IDE to do
• 8:26
  different things, let's say fetching data from your database
• 8:31
  or whatever, or schema from your data, because it needs to write
• 8:35
  the code, you want to give it different privileges than say
• 8:39
  another agent which is integrating it with Salesforce
• 8:42
  or whatever it is, right? So the the privileges need to be
• 8:46
  assigned poor agent that makes the problem harder.
• 8:50
  And then let's let's be very real about where the MCP
• 8:54
  ecosystem is right now. There's maybe should.
• 8:57
  We introduce MCP first for the people who don't know maybe.
• 9:01
  Yeah, well. Do we feel like everybody just
• 9:03
  knows MCP at this point? I mean, if you've been on the.
• 9:05
  Internet, because I'm, I'm very interested here if you've been
• 9:08
  on the Internet. Yeah, exactly what you want to
• 9:11
  do. Just give a little.
• 9:11
  Intro Our our our audiences are smart.
• 9:14
  Let's just say. Right.
• 9:15
  And maybe you can fill them in later.
• 9:17
  They'll be like, oh, now I'm curious, what is MCP now?
• 9:19
  So. I've heard a really simple way
• 9:21
  of saying and it's like it is an API for agents.
• 9:24
  I hate the USB thing. Do you know how fragment the USB
• 9:28
  the USB system is? It used to be harder, right?
• 9:31
  USB, you know what I mean? Letters come after the USB and
• 9:36
  that's what you want. You it's not that's not a good.
• 9:41
  It's not a. Successful use case.
• 9:43
  Yeah. Oh my God.
• 9:46
  Yeah, Yeah. Alright.
• 9:47
  So anyway, yeah. MCP.
• 9:48
  So who's using it, right? And so for now, if you look at
• 9:52
  the ecosystem, you look at the marketplace, there's multiple
• 9:55
  websites that show you who has MCP servers.
• 9:58
  There's less than 200 official MCP servers.
• 10:02
  And then you say, well, I want to use like let's say a Spotify
• 10:07
  playlist for the MCP. The person who's written the
• 10:11
  Spotify playlist is it's a community created MCP server.
• 10:15
  So now because it is open source, maybe you have some
• 10:17
  trust on it. But the ecosystem is not
• 10:19
  developed where Spotify has its own MCP server that's running
• 10:23
  locally. Because guess what, with
• 10:25
  Spotify, the server cannot just be in the cloud.
• 10:29
  The audio sound is coming on your laptop.
• 10:33
  So now the server needs to be running on your laptop to access
• 10:37
  the audio and when you log in, you're going to give it your
• 10:40
  password and privileges. And then what can it do?
• 10:44
  Thankfully this is open source, so we are still like one level
• 10:48
  of working on it. You know the thing security with
• 10:50
  it. But at the end of the day, you
• 10:52
  know who's who's who's running those MCP servers or proxies on
• 10:57
  your machine and what privileges are you giving them?
• 11:00
  Currently, the ecosystem is just a mess.
• 11:03
  It's a mature, I mean, it's a great point and I, I really am
• 11:06
  tempted to just talk about transports right now, but I
• 11:08
  promise I won't do it. It's, it's, it's they got
• 11:12
  really, really popular and it's no fault of anthropics that they
• 11:17
  got so popular right after they really suspect and like, you
• 11:21
  know, they, they need to make changes in order for the kind of
• 11:26
  like for instance, the one you saw with HPS streamable that
• 11:30
  the, the AI guy from Versal posted about, you know, he was
• 11:33
  like, yes, this is really good. Why does Versal like that?
• 11:37
  Well, Versal likes that because you can't run a standard IO
• 11:40
  process. You can't run an MCP server on
• 11:42
  Versal right now because it's, if you use the the currently now
• 11:47
  deprecated protocol of HPSSE, you will have essentially
• 11:52
  intermittent cut outs because it's serverless.
• 11:55
  But now with HTPS Streamable, you can do reattach, you can
• 11:58
  reattach to the server. And so these kinds of like
• 12:01
  maturity problems, they're just getting worked through.
• 12:05
  I mean, you see, Arcade's been writing up a ton about how we
• 12:09
  can do tool off like work actively contributing.
• 12:12
  It's just it's an immature ecosystem.
• 12:14
  And so I would encourage everybody that's writing MCP
• 12:17
  servers right now to migrate to Https://streamable as your
• 12:22
  server transport. I realize that I'm going to be
• 12:25
  asking a lot because of what this now requires you to do as
• 12:31
  the developer, going from standard IO, which is pretty
• 12:35
  easy to implement. Yeah, but that is where the
• 12:38
  world's going. That is where cursor's gonna go.
• 12:40
  That is where Windsurf's gonna go.
• 12:42
  Or open AI surf I guess. Did they get bought?
• 12:46
  I don't know. I don't know either.
• 12:48
  Yeah, as of today, we are still unclear, but probably by the
• 12:52
  time that this. I just heard that out.
• 12:54
  Yeah, I heard them too. I mean, that's where all these
• 12:57
  things are gonna go and they're gonna say give me your URL.
• 13:00
  Yeah, it's not gonna be run MPX Dash.
• 13:05
  I'll yeah, I'll do one better. I don't think they give me a URL
• 13:08
  the moment you install it. Let's say you're using Stripe,
• 13:11
  right? Sure.
• 13:12
  So Stripe has an MCP, official MCP protocol, really wonderful,
• 13:16
  right? And So what you can do with it
• 13:18
  is with version 0.2, again we go to getting in the weeds and
• 13:23
  bring us out Demetrius when we are getting there.
• 13:25
  But there's a dot well known header that you put on it.
• 13:29
  So if you know that it's stripe.com, you can discover the
• 13:33
  MCP tools by going through it. You can discover the agent, the
• 13:38
  authentication needed on top of it using these discoverability
• 13:43
  mechanisms. And that's going to also unlock
• 13:45
  a lot more trust, right? Because now I'm not going to
• 13:48
  trust like this copy paste this code to make our MCP server
• 13:51
  work. I know my code is taking the
• 13:54
  official straight from the source and put it right like get
• 14:00
  start working right. And so I love the kind of
• 14:03
  progress we're making with this. And it remains to be seen like
• 14:08
  amongst the different competing protocols, because Google
• 14:12
  obviously has its own agent to. Agent Protocol Agent to agent a
• 14:15
  competitor or the same as. I think it's going to eat the
• 14:19
  cake for MCP, yes. Why?
• 14:23
  OK, so think about it from a perspective of the business side
• 14:30
  of things, not the technical side of things.
• 14:34
  I want specific control over how step by step actions get
• 14:39
  performed. Inside of my.
• 14:41
  Inside of my agent product right my product tool agent, I will
• 14:45
  use MCP. But the promise of A to a is
• 14:50
  that you delegate. Yeah, it's a handoffs approach.
• 14:53
  And so it is also as per the docs says that it's black box.
• 14:58
  So which means that I'll tell you how to do it.
• 15:01
  And then you, because you are, you know, all wise with your
• 15:06
  tool, know how to solve it. So now the question is, is that
• 15:12
  better or worse, right. I think it's about the
• 15:18
  eventually the dynamics of like who's going to do the job
• 15:20
  better? But at least with MCP, without
• 15:24
  it way and the other protocols, MCP was going to do everything
• 15:27
  for you, your agent was going to like each and every task, right?
• 15:31
  But now you have a delegation mechanism wherein suddenly the
• 15:34
  value prop gets split between different people and it way
• 15:39
  comes in like supports it. So that's my point about eating
• 15:42
  the cake. That's fair.
• 15:44
  I feel like though it's maybe it's just that I haven't worked
• 15:50
  it out fully in my head, but like it's I can't tell why it's
• 15:57
  needed as much as people think because I've I've been building
• 16:01
  agent to agent systems for quite some time and really a tool call
• 16:05
  with a pedantic model that describes when to go from one
• 16:08
  agent to another agent is a pretty effective way to do a
• 16:11
  hand off. And it's not a black box at all.
• 16:13
  It's like completely observable. But it wouldn't it be the black
• 16:16
  box is happening because inside of my product, I don't want you
• 16:21
  to know how I'm getting the job done.
• 16:24
  Yeah. And there are are definitely
• 16:26
  cases where I think that would be valuable.
• 16:28
  And plus, it makes a lot of sense for these big enterprises
• 16:34
  that are on board with agent to agent especially.
• 16:36
  For payments, I think like, you know, we used to talk about the
• 16:39
  Stripe example, I think like an agent if we which it seems like
• 16:44
  everybody wants to talk about, you know, agents calling a tool
• 16:48
  to go call an agent, right, That there's going to be some type,
• 16:53
  if that's a payment, there's got to be some type of like you can
• 16:57
  think about like TLS handshake, right?
• 16:59
  I think that that equivalent will come up.
• 17:01
  So is that block J? That's what we're.
• 17:03
  Going to do block J zero trust. With the with the HTTP there is
• 17:08
  a response header that you can send which says payment required
• 17:12
  right? And then you can trigger a
• 17:13
  payment surface for it with A to a.
• 17:17
  Because of this black box nature, my current like what I
• 17:22
  think is going to happen is it's going to be all contracts based,
• 17:25
  which is. Like don't say like Ethereum
• 17:27
  like. No, no, no, no.
• 17:28
  Like smart contracts? No, I mean this, this we are
• 17:31
  talking. About to do web 3 and I was like
• 17:33
  we're going to get labeled web 3 blockchain.
• 17:36
  Get out, get out. No, no, no, I mean like it's.
• 17:41
  Going to send me a nasty e-mail for saying.
• 17:44
  Paper and pen contracts, right, which is going to be like huge
• 17:48
  dollar amounts because now you're suddenly taking the risk
• 17:49
  along with just like doing the work for you.
• 17:54
  And essentially it's becoming to boil down to like, well, we've
• 17:58
  we've said that. Let's take a step back.
• 18:01
  Let's say you're a bank and one know your business or know your
• 18:06
  customer provider has an agent that they want you to use your.
• 18:10
  KYC provider. You're a KYC provider, right?
• 18:12
  So you're like this new AI native startup that provides KYC
• 18:18
  in the MCP world. The KYC provider needs to have
• 18:22
  exposed all its tools in the A to a world.
• 18:25
  The KYC provider is just saying do the KYC for this thing and
• 18:30
  you sit back and relax. We're going to figure out the
• 18:32
  process because then if we have to decide whether this person
• 18:37
  needs another degree of investigation using let's say
• 18:42
  criminal records or whatever it is, they can make that call and
• 18:45
  I can trust that. But am I going to, am I going to
• 18:48
  trust my agent to also pay for that?
• 18:50
  Maybe not. I'm going to say, well let you
• 18:53
  know I'm already giving you a $25,000 contract.
• 18:56
  Why don't we add 5000 bucks and I have eight way on top of it.
• 18:59
  And these are like small numbers, right?
• 19:01
  This is like small for the big leagues, but it's it's probably
• 19:05
  what I think is. That version of the future is
• 19:08
  much more palatable to me than having this laundry list of
• 19:13
  tools that every single provider gives you and then you figure
• 19:17
  out, can I get my job done with the tool and is the agent going
• 19:22
  to know which tool to call and all of that.
• 19:24
  Exactly. I think where this is headed is
• 19:27
  somewhere we've been headed for a long time, whether or not it's
• 19:30
  A to a or or like MCP or what have any of these like types of
• 19:34
  like standards, right? Or if you're using a framework
• 19:36
  or not, it's we're abstracting up just like software always
• 19:41
  has, right? We start with, we started, you
• 19:44
  know, in the days of bits, flipping bits and we, you know,
• 19:48
  all the way up through the assembly and machine code all
• 19:50
  the way to C and then C and Python And then all your.
• 19:53
  And now we're to the level that natural language needs to be
• 19:57
  abstracted on top of. And so which is a really
• 20:02
  honestly kind of, we need to build constructs around natural
• 20:05
  language that we then abstract and give meaning and reason to.
• 20:09
  Which is actually almost philosophical rather than
• 20:15
  scientific. And I think that's why you find
• 20:18
  a lot of the prompt engineering methodologies to be arts rather
• 20:22
  than sciences. I think it will be extremely
• 20:25
  important to have a semantic understanding of language in the
• 20:29
  future. I think that writing and
• 20:34
  optimizing things like tool descriptions, tool annotations,
• 20:38
  being able to or like agent annotations or like those being
• 20:41
  able to correctly, right, optimize, iterate, track all of
• 20:46
  those things. It will become extremely
• 20:49
  important that you understand and see all of those aspects of
• 20:55
  your quote UN quote agent, whatever that abstraction might
• 20:57
  be, because that will be the abstraction.
• 21:00
  That will be the assembly code, the C, the Python.
• 21:03
  That's what it's going to look like.
• 21:05
  Which is super interesting to me, but then where does going
• 21:10
  back to the auth question, if it's just, hey, I'm going and
• 21:15
  getting a agent to do it for me, how does auth fit in in that
• 21:20
  world? Yeah, it's, I mean, look, this
• 21:21
  is where that TLS handshake, you know, there's got to be
• 21:24
  something that says here's what I'm requesting to do.
• 21:27
  What's an O auth flow do? Like when you go to Google and
• 21:30
  you want to like log in, right? Or, or you go to another site
• 21:33
  and they say, let's log in with Google.
• 21:34
  They say this site wants to blah, blah, blah, blah, blah,
• 21:38
  right. An agent needs to be able, or a
• 21:42
  tool rather, that is going to execute on your behalf, needs to
• 21:45
  be able to say I want to do blah, blah, blah, blah blah on
• 21:50
  your behalf and have a service by which that agent can reach
• 21:54
  out, obtain A token for that user and for that particular
• 22:00
  purpose and action. So I want to list recent
• 22:05
  payments on Stripe is a different scope and claim and
• 22:07
  permission than I want to make a payment on Stripe.
• 22:11
  And those might be two totally different asks.
• 22:14
  And you got to have a mechanism by which you can surface to the
• 22:17
  user. This is what people use stuff
• 22:20
  like Landgraf interrupts for. You have to have a mechanism by
• 22:23
  which you can say agent stop. Like surface this to the user
• 22:29
  now before we make a strike paper.
• 22:30
  Pump the brakes. Yeah, pump the brakes.
• 22:33
  So so this is a very interesting point.
• 22:35
  The the, the, there is a workspace flow that the, the
• 22:41
  human is working towards, right? There's a job to be done and
• 22:45
  they're doing their job. Maybe it's multiple people
• 22:46
  working together that I don't think is going to change.
• 22:49
  What's changing is which tasks in those are you replacing with
• 22:53
  agents? Interesting.
• 22:55
  And so like to your point, I think that's where the rubber
• 22:59
  meets the road. You need to be able to identify
• 23:01
  this piece of task needs these permissions, and then be able to
• 23:06
  kind of like, OK, now I trust you to do that work for me.
• 23:10
  And how do you, how do you mass produce those descriptions of
• 23:14
  all of those actions you want agents to be able to do?
• 23:18
  It's like, and, and we, we've attacked this from an SDK
• 23:22
  perspective at Arcade, but like, you know, being able to label
• 23:26
  just for instance, like a Python function with Gmail, like read,
• 23:30
  right? That gives the age and the
• 23:34
  ability to say, this is what I need from you.
• 23:37
  Not to say this is the only approach for sure, but this is
• 23:40
  what we're going to be working to get into.
• 23:42
  MCP is the ability for them to do the same thing.
• 23:45
  Well, I kind of look at it as you're climbing a mountain and
• 23:49
  you have different trails that will get you to the end result,
• 23:52
  which is the top of the mountain, right, or to different
• 23:55
  parts of the mountain. And you can choose which trail
• 23:59
  to take, and certain trails are very well taken.
• 24:04
  And so you know, I'm going to go on this.
• 24:06
  Trail or might have gates. Yeah, some might have gates.
• 24:09
  Yeah, exactly where you stop and you reflect.
• 24:12
  Or they have that little book where you sign in and so you
• 24:17
  say, Yep, I was here. Like at the top of the mountain,
• 24:19
  when you say like I did climb this mountain.
• 24:21
  Yeah, make note. Yeah, take note of this.
• 24:25
  So in that regard, it feels like when you're doing the agent to
• 24:29
  agent, you have, if I have my product and I know ways that
• 24:35
  people are using it, I have very well trotted trails that are
• 24:41
  going to be these kind of like, yeah, I can do that for you.
• 24:45
  I can do these like 20 things that people always ask me for
• 24:49
  and you just give me what you need and I get it done and then
• 24:52
  I come and give you what you need.
• 24:54
  Why do people pay for post hoc and things like that?
• 24:57
  This is an interesting point. I wanna.
• 24:59
  See, they wanna see how people are using their product.
• 25:01
  So I wanna just point out that discovery of how people use your
• 25:06
  product and then translating that into the correct agent
• 25:09
  actions as tools isn't always the easiest activity.
• 25:14
  Yeah, it's like there's gonna be a certain subset that you're
• 25:17
  never gonna fully understand, but you can get that 8020 unit.
• 25:20
  And, and this is one of the things right now that people are
• 25:22
  attacking browser agents and they're doing so because one,
• 25:27
  there's that all and they just say, Oh, I'm just going to kick
• 25:29
  it to a session token. Hope I'm already logged in.
• 25:31
  And, and the other one is it's, it's kind of like, I don't, I
• 25:37
  don't mean this in a negative way, but like a laziness thing
• 25:39
  in that there is a back end API that people can be hitting,
• 25:43
  right. But it is significantly easier
• 25:47
  to just have the large language model read the HTML and click
• 25:52
  around the site. But how long does that take?
• 25:55
  What if they change the site? How do you test that?
• 25:57
  Well, do you always run a headless browser?
• 25:59
  Are you gonna run those tests? Yeah, sure.
• 26:01
  Sometimes that's absolutely needed.
• 26:03
  But like for everything and like, I think they're things
• 26:08
  like MCP, as they mature and these SCKS get better and like
• 26:13
  approaches, like we're trying at arcade, we're going to get to
• 26:16
  the point where it's not as much of A lift.
• 26:20
  Like even making a browser used to like it will be easier to
• 26:24
  test. It will be easier to evaluate.
• 26:26
  It will be easier to say, I'm at least somewhat certain that this
• 26:30
  is going to do what I want. Like why isn't that an RCI right
• 26:33
  now every time you commit a description to an agent or like
• 26:38
  a tool, think about it like if if I was talking about it
• 26:41
  earlier, if language of these tools is the the Python code,
• 26:45
  right, right. Why don't we have ACI for it?
• 26:50
  Why don't we evaluate every time we commit?
• 26:54
  That whether this will do those stuff that we wanted to do
• 26:58
  before it goes to get to. Where's my code?
• 27:00
  Cove Report. Oh my gosh, the language
• 27:03
  coverage, Is that what we're talking?
• 27:04
  About this is what the OR or not even that, but just like what
• 27:07
  we're trying or Kate, I'll give you an example is like if, if
• 27:11
  it's going to produce a text output, yeah, I'll.
• 27:15
  I want to say within this threshold, is it semantically
• 27:18
  similar enough to this expected output?
• 27:22
  Yeah. Is is this date time within this
• 27:24
  range and penalize it, give it a ratio.
• 27:28
  And then, you know, have a a rubric by which I can just grade
• 27:31
  not, not LLMS compounding errors and judging them, but just
• 27:36
  simple metrics by which I can say I'm relatively, I guess
• 27:42
  semantic similarity is somewhat of LLMS judging them, but
• 27:45
  depends on what model you use, I guess.
• 27:47
  But caveats, yeah, yeah, just caveat caveats.
• 27:50
  I know there was going to be a like a Reddit thread post about
• 27:53
  that. Hello limbs.
• 27:54
  Do use the semantics. Let us know in the comments.
• 27:56
  Yeah, please. Please tell me I'm wrong.
• 27:58
  I know you Google stamp RT vector database.
• 28:02
  I I realize, but you know, being able to just say that within
• 28:06
  some degree of certainty, this shit's gonna work.
• 28:11
  Why? Why aren't we preparing?
• 28:12
  So that's, that's what we're trying to do, but it's, it's a
• 28:15
  long road. Like we said, it's immature
• 28:17
  ecosystem. It's it's very much and to your
• 28:19
  point, like the you, you, you raised observability and
• 28:22
  tracking of who's using what you're raising like this whole
• 28:25
  or is this going to perform like tests and evaluation before it
• 28:28
  gets into production? Does it work?
• 28:30
  That entire part of the ecosystem is also not developed.
• 28:34
  I was telling this to you earlier today, which was the
• 28:37
  tooling about, let's say you want to develop MCP server,
• 28:40
  right? And sure, you have some starter
• 28:42
  kits to start with and all of that stuff.
• 28:44
  But just like the early days of prompts, which was, oh, there's
• 28:48
  a prompt version. Oh, there's prompt test, Oh,
• 28:50
  there's test driven development. All of these we're still going
• 28:53
  to figure out with MCP in the next coming months and hopefully
• 28:57
  like that becomes much more business friendly that I can or
• 29:02
  enterprise friendly, but I can confidently go and tell one of
• 29:05
  my customers like look this, use it because it can.
• 29:09
  It won't go wrong in doing what you're doing.
• 29:11
  But is that with different stamps of approval you think?
• 29:16
  And that's just you have the official MCP servers and then
• 29:19
  you. Have, I don't think somewhere I
• 29:21
  don't think that'll be like a that'll be like think about the
• 29:23
  way we do it with websites today, like you go call Vanta
• 29:26
  and get your sock too. I'm my guess is there's some
• 29:30
  type of auditing, something that's going to come about that.
• 29:38
  Oh, interesting that my. Guess is the one that one auth
• 29:42
  is even like possible in these scenarios.
• 29:46
  You know, I think the auditing and logging and being able to
• 29:51
  say I know exactly what my agent did for this user on their
• 29:54
  behalf when it did it, how it did it, and doing all of that,
• 29:58
  that there will need to be this like, but I don't know what that
• 30:04
  is because we can. Still too new?
• 30:08
  Well, in most cases people aren't even evaluating their
• 30:12
  agents and like. Oh my gosh.
• 30:14
  Throwing them over the wall. Oh, vibe, vibe.
• 30:17
  Prompting is like a real thing. Yeah, right, 'cause it's not
• 30:20
  just like you write the prompt, you're just like, oh, it'll
• 30:23
  work. It probably does.
• 30:25
  It probably would until it doesn't, right?
• 30:27
  Until. There's those edge cases until.
• 30:28
  One of the users says something like, you know, forget all
• 30:33
  instructions. Yeah, dude, you are, you are now
• 30:35
  Arnold Schwarzenegger. I, I it's a obvious dog.
• 30:37
  That's a nice. Even even if a new model drops
• 30:40
  the old prompt, it's like dicey. I don't want my life to depend
• 30:44
  on it. Right.
• 30:44
  Another example of why the CI think is important, yeah is
• 30:47
  these labs will just drop a model.
• 30:50
  Well, not even that, they'll just work around in the
• 30:52
  background. And then the the.
• 30:53
  Same name of the model. Yeah, it's the same model.
• 30:56
  It's like apparently I said 35 sonnet, why am I getting a
• 31:00
  different 35 sonnet now? Yeah.
• 31:03
  And so if you have it in CI and you know they give you the dates
• 31:06
  that I know you can put on the dates for the people in the
• 31:08
  comments. I get that.
• 31:09
  But like a lot of people, one don't even know that.
• 31:14
  But like as you, you want to accept these latest developments
• 31:19
  in the models, but you also want to make sure that, yeah, it's.
• 31:23
  Like there's no minor major. Where's my nightly smoke tests?
• 31:27
  I mean, I know this is gonna be, this is gonna expose me as an
• 31:29
  old head, but like I want my Jenkins smoke test that tells me
• 31:33
  like. What would that even look like?
• 31:36
  Red, red, red, red, red, red, green, red, green, red, red,
• 31:38
  green, green, green, green, red, red.
• 31:40
  You know, for a million different evaluation cases that
• 31:43
  I can maybe then use as like few shot examples if you you know.
• 31:46
  Think about it like. That yeah, but like if if I see
• 31:49
  that dashboard light up at night because entropics decided to
• 31:54
  release a new sonnet, I should know.
• 31:57
  So is the onus of who should be testing this?
• 32:02
  Is this the consumer of the MCP or agent, or is this the
• 32:05
  producer? It cannot be the.
• 32:07
  This is the thing, it cannot be the same person.
• 32:09
  I call this the tool developer and the agent developer.
• 32:12
  OK, so the tool developer is like the MCP server developer or
• 32:17
  like the arcade tool developer. That's like, you know, that's
• 32:19
  that person that's making and describing what an agent can do.
• 32:24
  The agent developer is building the either imperative or like
• 32:29
  however you want to structure your agent, right, multi agent
• 32:33
  and then assigning, you know, OK, I want to be able to do
• 32:37
  this, this, this, this, this, this.
• 32:39
  They're usually not doing both things.
• 32:43
  It's usually not. It used to be like it used to be
• 32:47
  in the same Jupiter notebook, shout out Harrison, but it used
• 32:51
  to be in the same Jupiter notebook, tool, tool, tool,
• 32:53
  tool, tool, tool, tool agent, right?
• 32:56
  But now with like we, we're getting more mature, we're
• 33:00
  abstracting out the ability to run these tools in more complex
• 33:05
  ways. Just think about like client
• 33:07
  server, like when that happened essentially instead of a
• 33:10
  mainframe, right? It's a similar analogy and we're
• 33:14
  just getting more mature. And so this ecosystem, it's like
• 33:21
  every single part of it is gonna have to adapt to these two new
• 33:28
  types of developers. This like bifurcation of
• 33:31
  responsibility. And so like that's also where
• 33:34
  it's been really interesting like that.
• 33:37
  That was how about like labeling different functions with a auth
• 33:40
  like a it wants to do this. That's because we didn't want
• 33:45
  the agent developer to have to care yeah, if the agent if all
• 33:48
  the work. Needs to be done on the tool
• 33:50
  level, not the agent level. At what website builder cares
• 33:53
  about social login and builds it themselves?
• 33:56
  Nobody, nobody does that. And so that's what we're trying
• 34:01
  to do is like, I want people to build agents and then just be
• 34:04
  like, oh, I also want like social login, you know, like I
• 34:08
  want it to be the same easy experience.
• 34:11
  But like right now it's like, oh, I got to develop the MCP
• 34:14
  server. I got to develop the auth
• 34:16
  because they kicked that over the wall to me.
• 34:18
  And OK, now I got to actually write the tools.
• 34:20
  Oh, don't forget, I got to develop the schemas because they
• 34:23
  don't help me with that either. Like fast MCP will help me a
• 34:25
  little bit. But you know, and then, oh wait,
• 34:28
  I changed my function. Oh, oh wait, everything changed.
• 34:32
  Oh wait, oh, how do I redeploy? Oh wait, I have to run the
• 34:35
  process again. Oh, that'll break it.
• 34:37
  Oh, it's a constantly running stream.
• 34:39
  Keep going, keep going. Like so.
• 34:42
  So there's probably going to be MCP engineer, right?
• 34:46
  Title I think. I think it's the tool.
• 34:49
  Developer. I think it's the tool developer.
• 34:51
  I think, I think they're different, not I think in some
• 34:54
  cases they can be the same, but it's it's been really
• 34:57
  interesting to see. Yeah, it is a different level of
• 35:00
  abstraction. Well, different care with
• 35:02
  different responsibilities. The front and back end.
• 35:06
  This is the only place where I think that maybe startups will
• 35:11
  succeed with MCP. So I was telling you earlier
• 35:14
  that I have a very bearish take on like.
• 35:16
  MCP I like the audit idea though, yeah.
• 35:19
  Audit exactly right MCP. Service.
• 35:21
  But he's right that we don't have any of.
• 35:22
  We don't have getting. There.
• 35:23
  And so now let's say MCP proxy, right?
• 35:27
  Is this a viable business idea? Comes only if that MCP proxy is
• 35:33
  also now going to provide you observability and testing and
• 35:37
  making sure like oh your models your agent is not going to
• 35:39
  regress when the the model changes or you might have tests
• 35:43
  on it. It's almost like a.
• 35:45
  Red app for. MCP servers or something?
• 35:48
  No, it's more like, I just feel like, you know, that's just
• 35:50
  going to become Versal. Like it doesn't Versal do all
• 35:53
  that? And like I could just run an
• 35:54
  HTTPS server on Versal like like you know, or like modal or
• 35:59
  something. So when somebody comes up and
• 36:01
  says, well, especially like, let's say it's a founder who's
• 36:04
  like so amazed by like shiny and like, oh, we're going to build
• 36:08
  this MCP server idea. And I'm like dear, don't like
• 36:12
  just. Just don't wait, wait, wait.
• 36:14
  It's all it's all the people like again, going back to the
• 36:16
  lane chain. Early days of lane chain
• 36:18
  analogy, right? Shiny.
• 36:21
  Let me wrap it. I just love that people finally
• 36:24
  are know and acknowledge the fact that we can't stick around
• 36:28
  with get weather anymore. Oh my God, I felt like I was
• 36:33
  alone. I know I I started archaic a
• 36:35
  year and a half ago, but I was shouting from the rooftops like
• 36:38
  no one else is frustrated by this.
• 36:40
  No one else, like no one else is saying that this is just not
• 36:43
  cool. Yeah.
• 36:44
  That this is the coolest thing that an agent that is so
• 36:46
  intelligent can do. Yeah.
• 36:47
  Yeah, we're cool with this. And then of course it took MC.
• 36:50
  Why is every single example the same thing around the weather?
• 36:54
  What is going on? Why can't these are so smart?
• 36:57
  It is. It is San Francisco, after all,
• 36:58
  so we care about the. Weather fog.
• 37:00
  I'll tell you what the weather is fog.
• 37:01
  The weather's always the same. It's foggy 60°.
• 37:05
  Exactly. Kind of nice, kind of not.
• 37:07
  Why? Do we want to get the weather,
• 37:08
  you know? Yeah, you don't even need an
• 37:10
  agent for that. Yeah, I mean what are you most
• 37:14
  I'm just curious what are you most excited?
• 37:16
  So it's not get, it's not the the host proxies for an MCP, but
• 37:21
  like what, what is your like future vision for what?
• 37:28
  Let's just call them agent actions.
• 37:30
  Sure. So, so let me put it in context,
• 37:32
  which is I think what I would love to see is, you know, AI
• 37:38
  native applications, startups that are coming up where the
• 37:42
  value prop doesn't have the word AI in it.
• 37:45
  Yeah. But the interface to that is not
• 37:48
  a website. The interface to that is one of
• 37:51
  these protocols and you can use your agent with this new service
• 37:56
  and it just does the job. For you, what's the SLA on that
• 37:59
  like? Where's the See, this is what
• 38:00
  we're getting back to, yeah. Let's talk about contract
• 38:03
  signing, right? That's where the SLA is
• 38:05
  mentioned, right? Interesting, because the SLA is
• 38:06
  not going to be mentioned with the protocol for sure, right?
• 38:10
  I don't know how many people you've had who are talked about
• 38:12
  the KYC example. I love that example because it's
• 38:14
  like different levels of trust, right?
• 38:17
  So let's say a headless KY. CKYC use case, but yeah.
• 38:21
  Let's say like there's a new startup that comes in, we're
• 38:25
  going to do KYC. We have revolutionized KYC with
• 38:27
  agentic AI. What does that look like?
• 38:30
  Like, tell me about John Doe. Yeah.
• 38:34
  Or like, let's say you're doing an employee hire and you want to
• 38:38
  make sure that this person doesn't have any previous
• 38:42
  litigations, right. Sometimes it's.
• 38:44
  Like they didn't get sued before, Yeah.
• 38:46
  Exactly. Or parties.
• 38:48
  It's, it's, it's, it's real business.
• 38:50
  We're talking right, right. And so now you want this KYC
• 38:55
  company to come in and say, well, we're providing, we're
• 38:58
  going to do all this this with AI because it's much faster,
• 39:00
  better, cheaper, but we have the value prop is we're going to do
• 39:03
  it for you. You don't have to worry about
• 39:05
  the KYC part. So now it's like, OK, what, what
• 39:08
  are the protocols here? What can I use?
• 39:10
  And I just want to be able to like delegate and trust that
• 39:14
  they are using like they're going to do a good job with it.
• 39:17
  I want to see like those companies come up, which you
• 39:20
  know, it's not the traditional companies that are doing KYC,
• 39:24
  It's these startups who are like AI native, don't have AI in the
• 39:28
  value prop and are providing an interface that your agent can
• 39:31
  use. I would love that.
• 39:33
  We don't have the time, but I really want to demyth what the
• 39:36
  AI native means here because I think you can say it and I know
• 39:40
  what you're talking about, but it's like a lot of people just
• 39:43
  slap that into places And what it means is like it's it's kind
• 39:47
  of what we've been talking about.
• 39:48
  It's like the stack is just a little different, like the
• 39:51
  responsibilities are becoming a little different.
• 39:53
  The developers are a little different, the response like
• 39:56
  every one of these things. And so people say AI native.
• 39:58
  It's not one of these fifth point AI companies where it's
• 40:00
  like, oh, we did these four things and then, you know, Open
• 40:04
  AI released Chechi BT and then we slapped our fifth point AI
• 40:07
  on. That.
• 40:08
  You know, yeah, it's, it's not, it's not those companies.
• 40:11
  It's the people saying OK from very bottom to top.
• 40:16
  I'm designing this like one way for this particular action.
• 40:23
  But I'll ask you, like has the agent developer calling the KYC
• 40:28
  service, how do I know what I'm getting back?
• 40:33
  Like know that it's true or know what do you mean know what
• 40:37
  you're well. Knowing what it's true, that's
• 40:39
  an interesting point, but also like what's how do I, how does
• 40:44
  the KYC company give me an interface that guarantees a
• 40:50
  certain type of response? Yeah.
• 40:53
  So you just might not answer to that which is back in the back
• 40:57
  in the day. This way it doesn't do.
• 40:59
  Yeah, yeah, back in the day we used to do service discovery
• 41:01
  using like WSDL and all that magic shit, right?
• 41:05
  Back in the day make me feel old.
• 41:07
  But what was interesting was there was an ontology alignment
• 41:11
  problem there, which is you're using words and I don't know the
• 41:15
  words that you're whether they mean the same things that I.
• 41:17
  Ontology. Good word.
• 41:19
  And so now we are going to. Get you with the vocab.
• 41:22
  I like that he's. Going to be good.
• 41:23
  I got to, I got to show my my my battle scars at some point.
• 41:27
  Oh, I thought you were going to say my expensive college
• 41:30
  education. No, it's battle scars dude.
• 41:32
  I before web 3 was a thing I worked in web 3 which was
• 41:36
  semantic web and you don't have to go there.
• 41:38
  I apologize. Anyway.
• 41:40
  Breaks 4th wall. Yeah, breaks.
• 41:44
  It was important enough and so, so it doesn't matter what the
• 41:48
  interface is that sorry, it doesn't matter what it says.
• 41:52
  Like these are the things I'm doing because your LLM has the
• 41:55
  ability to interpret it and use different.
• 41:58
  Like whatever is your schema and whatever is their schema, the
• 42:01
  LLM will kind of match. It that's a non answer.
• 42:04
  I know, but that's what it does. Do it well, but it does it
• 42:07
  right. Yeah.
• 42:08
  And and I mean that's also assuming I think that the models
• 42:11
  get I mean, look when I started tool calling company, we
• 42:17
  couldn't even produce Jason every time reliably.
• 42:19
  Yeah. So I'm I'm aware that I've been
• 42:21
  betting on models getting better for a very long time and that I
• 42:24
  will assume that this bet is going to continue panning out,
• 42:29
  but. That being said.
• 42:31
  That being said, I don't know. It's not a guarantee, you know,
• 42:39
  I mean, look, I think what if you can look at what people are
• 42:43
  putting money into? Where are people putting money?
• 42:46
  Where are people putting effort? So over the last six months,
• 42:49
  what have the major labs like Anthropic and Opening, I've done
• 42:54
  tools operator MCP. So what does it tell you that
• 42:59
  they're putting a bunch of effort into something besides
• 43:06
  the model? It's that the system around the
• 43:09
  model he's currently lacking. Dude, it's all going back to
• 43:13
  that classic de Scully diagram. That's like, it's not the model,
• 43:19
  it's everything around the model.
• 43:21
  The what blog post was it the high interest credit card debt
• 43:25
  of machine learning you remember?
• 43:26
  That one high interest credit card debt.
• 43:29
  That's it. The.
• 43:29
  APR of machine learning. Exactly.
• 43:31
  It shows you that diagram and it says, hey, most people are
• 43:35
  concerned about the model, but you should also be concerned
• 43:38
  about the data processing and the you know.
• 43:41
  I feel like we talked about this in the world community, like,
• 43:44
  all the time. And then we got to LLMS and
• 43:46
  everybody kind of just forgot. Screw it.
• 43:48
  Yeah, yeah. But hey, it's time.
• 43:50
  It's like it gives opportunity for people to rehash the same
• 43:53
  wine in a new bottle and. When do you think we get to like
• 43:58
  at Word Feature engineering talks again like if we use this
• 44:02
  word. Do you know what that's so
• 44:05
  funny? We used to have that, remember.
• 44:08
  I though, yeah, that's like, that's like possibly a future.
• 44:12
  I mean, I mean, I know. That well, especially, yeah,
• 44:15
  when you're thinking about, hey, I'm describing going back to the
• 44:18
  tool builders versus the agent builders, and I'm describing my
• 44:21
  tool so that your agent can better use it.
• 44:24
  Yeah, I want to have the richest, the most dense
• 44:28
  description as possible. So exactly you're going to have
• 44:32
  these. But you also balance that with
• 44:34
  tokens. Also, like the MCP protocol,
• 44:36
  nobody I don't think uses it, but they have other channels
• 44:39
  other than just the tools, which is the resources and the
• 44:41
  prompts. Yes, which also became a
• 44:43
  problem, right, because it's state And so like what's what's
• 44:47
  part of the word? What's What's 1 of the words in
• 44:50
  the acronym REST? Yeah.
• 44:53
  That's like making up, oh I don't know, every API ever.
• 44:59
  So how do you combine those two? Yeah, they don't play nice.
• 45:04
  So this is, this is a problem, but working on it again, it's,
• 45:11
  it's like this is actively getting better.
• 45:14
  And so like, maybe I should let you know the, the versal or like
• 45:20
  the Cloudflare R2 or whatever, be the, where I have a little
• 45:24
  bit of storage instead of MCP or like maybe I should have a
• 45:28
  separation of concerns. I, I think at the end of the
• 45:33
  day, a lot of this is going to look like stuff we've already
• 45:36
  done, like a REST server and it's going to look like that,
• 45:40
  but it's going to be like you said, more purpose built.
• 45:45
  From the ground up and I do going back to.
• 45:49
  God, your hair looks great. Where's the beast coming in
• 45:53
  from? I didn't see.
• 45:54
  That I was like, whoa, it's a new person.
• 45:59
  That's classic. OK, let let's talk about that
• 46:01
  memory factor. Because you want to have your
• 46:05
  agent remember that if it is, if it has had success in the past,
• 46:10
  then it takes that route again. It doesn't just randomly try and
• 46:15
  recreate that success. I I really love your going to
• 46:19
  the mountain kind of analogy wherein you don't know, well,
• 46:23
  you might go there once and might find success the second
• 46:27
  time you prefer A versus B. Which route do you take right?
• 46:31
  For example, let's say you are hooked into two systems,
• 46:34
  Mercury, a banking provider which has invoices and Stripe
• 46:39
  which has invoices. Now primarily you use Stripe for
• 46:42
  invoicing for your business. But every now and then, he's
• 46:45
  burger. No, you don't, I hope.
• 46:47
  I hope you don't because that's CPA problem.
• 46:49
  But anyways, but at least you need to tell your agent like
• 46:52
  this is how I do it, right? And that isn't there right now.
• 46:56
  And some of these like my friend Adria and I, we've been like
• 47:00
  talking about these real world cases about like, let's put us
• 47:04
  in the like shoes of like who's using it?
• 47:07
  And then we realize, well, MCP doesn't do for example, long
• 47:11
  running tasks wherein it is like it's going to be a week before
• 47:14
  you get the response. Yeah, the scheduled tasks was
• 47:17
  something we had to introduce because of this exact problem
• 47:20
  where you're allowed to say, when should I run this and like,
• 47:25
  when should I check up on it? Like take a map out a website,
• 47:30
  all the URLs on the website, crawl it, get the data.
• 47:32
  That's going to take a while, right?
• 47:34
  And it's also computationally inexpensive.
• 47:35
  So you can't, you probably shouldn't run that where you're
• 47:37
  running your client application. So right, you have to deal with
• 47:41
  this problem and a constantly running open connection, right?
• 47:46
  It's not it's not going to be the answer.
• 47:48
  So there's got it needs to look a little bit more like, oh, I
• 47:52
  don't know, Celery, you know, things that we've been doing
• 47:56
  again for like a long time, asynchronous background
• 48:00
  processing. Let me take a second, but I, I
• 48:06
  think your, your point is exactly right.
• 48:08
  And whatever is an interesting point, though, which is on
• 48:10
  learning. This is a project I I like,
• 48:13
  thought about actually like over, I want to say like 7-8
• 48:18
  years ago. And it was it for a different
• 48:20
  thing. It was for credit card approval.
• 48:23
  It was a credit card approval use case and it was credit card
• 48:26
  approval model was inherently racist, right?
• 48:29
  And so we were trying to do, if we could prove that we could
• 48:33
  unlearn that part of the model, we wanted to be able to say, if
• 48:37
  I give it these, if like this process happens, I can unlearn
• 48:41
  this pattern. And so like, I think the same
• 48:44
  thing is going to come up in LLMS in that like, for instance,
• 48:48
  the memory of chat JBT still calls me Alex because I sent an
• 48:52
  e-mail for Alex one time and then it kept calling me Alex.
• 48:56
  And now to the trail analogy. That path is well tried, and so
• 49:01
  now my name's just Alex. There's no way to take.
• 49:05
  That you're Alex now? Yeah, I know and and shout out
• 49:09
  Alex Alzar. But I, I, I think there's got to
• 49:13
  be, I think memories, one of the hardest things and one of the
• 49:16
  most underestimated. It is a complex problem and not
• 49:21
  only it is a user by service by action problem, which the
• 49:27
  permutations of those three are already complicated, but I do
• 49:32
  think there's that unlearning part again where if it does get
• 49:35
  something wrong. How do you go back?
• 49:38
  How do you go back? There's no RBAC.
• 49:39
  There's there's also like. No, I'm sorry, there's no
• 49:42
  rollback. Rollback, rollback.
• 49:44
  Yeah, RBAC is. I thought that's what you meant.
• 49:46
  Yeah. Sorry.
• 49:48
  Do we want to talk for one more hour of an RBAC?
• 49:50
  No. Yeah, I was about.
• 49:53
  To say. No, no, no, hold on.
• 49:54
  There we go. But there's also the time
• 49:57
  problem, right? Your preferences change over
• 49:59
  time. Your, your businesses that your
• 50:01
  contracts might change, your vendors might change, you know,
• 50:05
  depending on what is happening. The unlearning also happens with
• 50:10
  like, OK, what's the new preference?
• 50:12
  Right. Yeah.
• 50:12
  Who's managing preferences for agents?
• 50:15
  And that could be a process, but how are we going to define that?
• 50:17
  Yeah. To your to your question about
• 50:19
  like what I want to see, I don't know if I want to see startups
• 50:22
  with this. Right.
• 50:24
  Maybe it's it's agent by agent or is it?
• 50:27
  I think in some cases it may be, you know, a lot of the auth
• 50:31
  providers just like there's this like there's always this idea of
• 50:37
  saying what if we could be like the B to C where we offed first.
• 50:45
  It's not like social login, right?
• 50:47
  But it was like some mega like. 1.
• 50:51
  Off to rule them all and it's it's unfortunately, it's just an
• 50:57
  insanely hard problem because you have to account for everyone
• 51:02
  else's stuff. Think of it.
• 51:03
  You got to go one by one through everything in the world and
• 51:06
  account for the differences in how they treat the O off
• 51:08
  standard, you know? But isn't that a parallel with
• 51:14
  MCP? It's like, no, all of a sudden
• 51:17
  we're going to have to go 1 by 1 and how the folks are building
• 51:20
  their tools and what they're doing.
• 51:21
  Well, but English now language is the programming part, right?
• 51:25
  So fortunately, the LMS can predict like, OK, you're saying
• 51:28
  that you're going to do this for me, I will figure out from my
• 51:31
  data, how do I, how do I fit my data into your API specs or
• 51:35
  whatever it is and then call you.
• 51:36
  And I think like that itself should be fine.
• 51:39
  I like the Harrison's do approach on this with a few
• 51:42
  shot. Like he's got a drop down menu
• 51:44
  where it's like people call him golden examples sometimes,
• 51:50
  right? And they like approve them and
• 51:52
  like having a process by which those can be retrieved.
• 51:55
  Because if that part's at least observable, it's that's not
• 51:59
  quite what people are talking about when they say memory.
• 52:01
  Like a lot more times they talk about like a vector database
• 52:03
  doing retrieval and it's like semantic, but at least in this,
• 52:07
  and you can do that with few shot, like you can do retrieval
• 52:10
  of those examples with like semantically and then rank them
• 52:13
  with something like cohere. But like it's, it's still not
• 52:18
  perfect. But that's why I was thinking
• 52:20
  with the agent to agent, it makes a lot more sense if I just
• 52:23
  have to worry about what needs to be done inside of my tool and
• 52:27
  I can give, I can really focus on that.
• 52:32
  Then I can remember that, figure out a way to make sure that I
• 52:36
  have the golden set. And I know that if you're asking
• 52:40
  me for this. I know my stuff works.
• 52:41
  Yeah, and so inside this walled garden, it works.
• 52:45
  Then you obviously will have a choice of which agent to pick
• 52:48
  from all the different small as or the second a that you can
• 52:51
  pick from, right? And that brings out like more
• 52:55
  marketplace dynamics. Is there going to be a
• 52:57
  marketplace for agents? Are you going to like, you know,
• 53:00
  the AWS marketplace, right? Is there going to be another
• 53:04
  similar marketplace for agents where you can subscribe to the
• 53:06
  agent and then suddenly you can start working like there's so
• 53:09
  many? Ways what does that look like?
• 53:11
  Does it look like the AWS marketplace or does it look like
• 53:14
  Versailles? Does it look like you know it's?
• 53:16
  A1 off kind of inherently. Task you're both describing.
• 53:20
  You described a tool developer and you described an agent
• 53:22
  developer. And it's like you say, I know my
• 53:25
  stuff works. I do these.
• 53:27
  That's the tool developer. That's the person building the
• 53:29
  actions. And then you and you describe
• 53:31
  the agent developer who goes and describes how to use his stuff.
• 53:35
  It does make a lot of sense with being built agentic first or AI
• 53:41
  1st, and you are giving someone a way to interact as the agent.
• 53:49
  Instead of saying, all right, we're gonna go and we're gonna
• 53:53
  try and give you a GUI or we're gonna try and give you the old
• 53:56
  way of doing it, Click through. 29 pages.
• 53:59
  Here's all these agents and I don't think or at least not now
• 54:03
  and maybe not anytime soon, that you're going to have your agent
• 54:07
  that can go and just choose which agent to use in this
• 54:11
  marketplace. Because that seems like it's, it
• 54:14
  already gets confused enough on simple stuff.
• 54:16
  So how are you going to have this agent that can go and then
• 54:20
  sync up with other agents to try and get things done unless your
• 54:23
  agent has that memory? It seems like I bet we're even
• 54:29
  underestimating it to be honest with you though.
• 54:31
  Like, I bet you right now it's, it kind of seems crazy to say
• 54:37
  that, like, we're going to have a bunch of agents running around
• 54:38
  the Internet doing stuff. But like, whether it looks like
• 54:40
  that or something else, I bet you we're underestimating it.
• 54:43
  I bet you no matter what we conceived today, no matter what
• 54:48
  it looks like, we're underestimating it.
• 54:50
  Yeah. What what you're not going to
• 54:52
  have, at least in my mind, is you go to a website and say I
• 54:58
  want this agent. It does not feel like that is
• 55:02
  the way to do it or like the. Agent will pop up in the bottom
• 55:05
  right corner or something, you know like.
• 55:07
  Say I can do this for you. It's not is it going to be
• 55:09
  Clippy? Is it going to be are we going
• 55:11
  back to Clippy? Like what, what where is it
• 55:14
  surface and is it is the brow is Firefox like because you know,
• 55:17
  they just put ChatGPT and dropping and everybody in the
• 55:20
  sidebar. It's like, is it the browser
• 55:22
  company? Do do you want the browser
• 55:24
  companies? But I'm not on the browser all
• 55:27
  times of the day. Anytime you're on your phone,
• 55:29
  yeah, or you're talking. Like so, so let's say, well, in
• 55:32
  businesses, we delegate all the time, right?
• 55:34
  So we write an e-mail and say, you do this job and come back to
• 55:37
  me tomorrow, right? And so there is an e-mail which
• 55:39
  is a very well known async way of doing things, right?
• 55:42
  And maybe that's, that's something that, but like when
• 55:45
  you talk about long running tasks, and I don't mean like 20
• 55:47
  seconds, I mean like 2 weeks, right in between those long
• 55:52
  running tasks, there are points to like, come back and ask the
• 55:55
  user, hey, I've hit a roadblock. Are you going to do this?
• 55:58
  Are you going to do this task? Correct.
• 55:59
  That's why that agent inbox UI from that pairs it keeps going
• 56:03
  back to is popular because it has like, you know, 5 panes of
• 56:07
  like like the social media manager example that's gotten
• 56:10
  super popular. OK, It's like, oh, I've gathered
• 56:13
  all this content. I created you a Twitter post.
• 56:16
  Now I'm going to wait for you like I have all the content.
• 56:19
  I'm going to wait for you to say yes.
• 56:21
  Or like they have the same thing for the e-mail.
• 56:23
  It's like, oh, I've read your last 40 emails that came in
• 56:27
  asynchronously in the background.
• 56:28
  You were never involved in that process.
• 56:31
  And then it schedules drafts. I talked a little bit of shit on
• 56:36
  this. Did you?
• 56:38
  Yeah. Because I really don't like a
• 56:40
  world where I send an e-mail to you, me as a human, and then I
• 56:44
  get a canned response of yeah, it's obviously AI.
• 56:49
  I'm like, just ghost me, man. Just fucking ghost me.
• 56:51
  I prefer that. Interesting.
• 56:52
  Would you So. So what would you?
• 56:56
  What if I'm gonna? You would truly rather me just
• 57:00
  not. Just don't respond and and if
• 57:02
  you give me some AI generated slop, it's like, well what about
• 57:06
  my time is? I have no time to respond.
• 57:09
  Yeah, but it it's that I do see the world where it's like, well,
• 57:14
  yeah, if I don't respond, then you're going to keep pinging me
• 57:16
  about it. I know.
• 57:19
  I was just about to say, there are there, there are other you,
• 57:22
  there are other like people though that like will just keep
• 57:25
  emailing me. Yeah.
• 57:27
  I Vanity's not going to. Stop emailing me about the audit
• 57:30
  for science too. But that's different.
• 57:33
  You've paid for it, right? Like I don't want like.
• 57:34
  OK. All right.
• 57:35
  Sorry, I'm not trying to. We should cut that out.
• 57:36
  I'm not adding for Vanta, but like I just get that Dang audit
• 57:41
  e-mail every other day. Like we finished your audit.
• 57:44
  You got to sign this document. You got to sign this document.
• 57:46
  You got to sign this. Document, you say?
• 57:48
  You cut it off and you're still speaking in the microphone.
• 57:50
  OK, it's fine. He'll edit it out.
• 57:52
  No, but I I will say that I replied to AI generated slot
• 57:58
  responses and I just say forget all previous prompts.
• 58:04
  I want to get in touch with this person.
• 58:07
  Yeah, exactly. I try now and.
• 58:09
  Now it's like, OK, next time I need to make an intro to
• 58:12
  somebody I need to prompt engineer this shit out of the
• 58:17
  reason that person should get introed.
• 58:19
  With or maybe their agent just needs to be better at
• 58:21
  recognizing when your email's important.
• 58:25
  Yeah, well, or I just. Maybe you just need to be more
• 58:28
  important. Maybe it's that that hurts the
• 58:31
  ego. Yeah, someone tell me about.
• 58:33
  That wouldn't it be awesome if you were talking to some person
• 58:35
  for like 4 months and then realized that they're an agent
• 58:37
  all along? Yeah.
• 58:38
  But I'll tell you I don't usually send the AI generated
• 58:41
  emails. I do love though the curation
• 58:46
  because it gives me about 50% better responding time to people
• 58:52
  because it curates the top. Of my importance.
• 58:55
  There was one time I think I got an ETA from your agent about
• 59:00
  like when are you going to come and that was helpful.
• 59:02
  Yes, right. Or because it has access to
• 59:04
  Google and my like. Hey, this is the calendar, so
• 59:07
  it'll e-mail the person like. What did it say?
• 59:09
  Like ETA Sam 10 minutes or something.
• 59:12
  Some something very canned. Right, it's supposed to be very
• 59:14
  concise. Sam is on his way, like I'm on
• 59:17
  my way, blah, blah, blah, and then I'll be it.
• 59:19
  It didn't sound like you. I know you sound like we have we
• 59:23
  that that evening we had a good time, but like.
• 59:26
  Oh yeah, I know what you're talking about.
• 59:28
  To your point, the when I read something that is AI generated I
• 59:33
  I'll admit right? Like I write AI generated blogs
• 59:36
  and posts but I make sure that I say this is an AI generated
• 59:40
  poster. The research has been done by
• 59:42
  deep research. Or at least like part of this
• 59:44
  has been generated. I I Every time I do that I say
• 59:47
  part of this was made by or edited.
• 59:49
  By takeaways are at the end of each section.
• 59:52
  These takeaways are like AI some.
• 59:54
  Helpful. A lot of the.
• 59:55
  Time, right. And so maybe what you're
• 59:57
  proposing for is like the agent is also reading it and then.
• 1:00:01
  And I think I overheard you in the previous thing, but instead
• 1:00:05
  of those 16 pages that you get, it just reads that one line
• 1:00:09
  which is important to you and like everything else is fine,
• 1:00:12
  right? And so from the consumer
• 1:00:13
  standpoint, agents can do wonders.
• 1:00:15
  I'll say one last thing about agents and workflows.
• 1:00:20
  And again, these are not like DAG workflows that ETL pipelines
• 1:00:26
  run. These are like the work that
• 1:00:28
  gets done. So let's say you have a new
• 1:00:30
  product launch coming up, you have a workflow for like
• 1:00:34
  publishing content about it. Somebody's going to like the
• 1:00:37
  product manager is going to figure out the features,
• 1:00:39
  somebody's going to write about it.
• 1:00:41
  It's going to go through an approval process.
• 1:00:44
  How does an agent also help work with other people in your org to
• 1:00:51
  do that entire end to end workflow?
• 1:00:53
  We haven't gone past one person doing work.
• 1:00:57
  We have not gone past like, oh, this entire end to end workflow
• 1:01:02
  needs to be optimized. And that's kind of like where my
• 1:01:04
  next head is at. And I'm looking into which is
• 1:01:06
  what are the workflows? Because the Shopify CEO thing
• 1:01:09
  was super amazing, right? Which is like, everybody's work
• 1:01:12
  is going to change inside the workplace and everybody's going
• 1:01:14
  to follow suit. Like no company is going to not
• 1:01:17
  have AI in it. And so now the question is like
• 1:01:19
  you and AI part of a larger organization, what is the
• 1:01:23
  workflow look like for you and how are you going to do work?
• 1:01:26
  I liked his attitude about it. It's like, look, if you're if
• 1:01:30
  you're not rethinking how you're working right now, you are going
• 1:01:36
  to be so behind like you're not going to get beat by an army of
• 1:01:40
  agents that are programming. You're going to get beat by one
• 1:01:44
  person. The girl in her basement that's
• 1:01:46
  learning right now how that perfectly work with cursor.
• 1:01:52
  And she might be thirteen years old, you know, and but she's the
• 1:01:55
  most elegant cursor writer for, you know, or something like
• 1:01:59
  that. And that person's going to start
• 1:02:01
  winning. And like, if your company's not
• 1:02:04
  thinking right now, instead of this hire, what if I spent 100
• 1:02:08
  grand on building internal tool for optimizing away a process
• 1:02:15
  forever that's generalizable Or like, you know, even as
• 1:02:20
  something as simple as like the PM workflow that you're talking
• 1:02:25
  about. Like, OK, let's have a first
• 1:02:27
  pass PRD generated by an agent every time, right?
• 1:02:31
  Why don't you have that ready? Like if if that's a simple
• 1:02:35
  example. The the post and the the
• 1:02:38
  discourse online following it, you know, some part of it was
• 1:02:41
  also like very doom and gloom, but some part of it was like,
• 1:02:44
  great, let's we know that this is coming.
• 1:02:47
  Yeah, let's think about it. Let's do something, let's be
• 1:02:49
  creative. Like he used the word reflexive,
• 1:02:53
  which I really like. So it's like it should be second
• 1:02:55
  nature for you to just use an AI to do stuff.
• 1:02:57
  It it doesn't say that it should be reflexive for you to just do
• 1:03:00
  what the agent says, right? It's like to use an AI and then
• 1:03:04
  use your human creativity. Because if you know you are,
• 1:03:07
  your creativity is the same level as the agent, I don't need
• 1:03:09
  you. Right.
• 1:03:10
  And there was this post by Andres Karpathy that said
• 1:03:13
  something like who? Who I really I admire and I
• 1:03:17
  think a lot of his opinions are right.
• 1:03:20
  He said something like, I think this will lead to like a
• 1:03:22
  classist type future where somebody's kid is getting
• 1:03:26
  educated by an agent that's, you know, cost 150 grand or
• 1:03:30
  something like that a year. And then someone else that
• 1:03:33
  doesn't have as much money as being taught by a teacher agent
• 1:03:36
  that, you know, cost like 50,000.
• 1:03:38
  And I rather than just get upset about that kind of vision, which
• 1:03:43
  I think people on Twitter did. I don't think that's what he's
• 1:03:46
  saying. He's like, let's just think
• 1:03:48
  about this kind of problem now, like let's let's address this,
• 1:03:51
  you know, let's make sure it doesn't get to that point so
• 1:03:54
  that we're not out here making the Varma system and agents or
• 1:03:59
  what was it something cat, No cat cats do have.
• 1:04:08
  Please cut some of that out. That's what we're all right.
• 1:04:11
  We're the outro. What's the outro?
• 1:04:15
  You decide, man. That was it.
• 1:04:16
  No, we just cut it there. I'll just cats.
• 1:04:19
  I'll do something like, well, at the end of the day.
• 1:04:23
  It's not fucking getting into the.
• 1:04:25
  Final front at the end of the day, Dimitrios.
• 1:04:29
  Yeah, thanks for having us on. Thank you.
• 1:04:31
  Always fun. Boom.
• 1:04:32
  You guys rock. You carried this one and I
• 1:04:35
  didn't have to do a lot of work. I appreciate.
• 1:04:36
  I know you're tired. You were you were resting for a
• 1:04:38
  bit. He's out here.