Latest / Agentic Conversations (formally mlops.community) / LLMs at Scale: Infrastructure That Keeps AI Safe, Smart & Affordable // Marco Palladino// # 341

Transcript

LLMs at Scale: Infrastructure That Keeps AI Safe, Smart & Affordable // Marco Palladino// # 341 · Agentic Conversations (formally mlops.community)

0:00
The way we're using third party software or APIs or services is
0:04
going to fundamentally change. I do believe and I truly believe
0:07
that the agents or even the prompts is the new browser.
0:11
You know, we're not going to go on a browser, click around, you
0:14
know, do things like we were doing in 1999 or 2022.
0:18
But we're going to be leveraging agents to do their job for us.
0:21
And when there is an agent on the other side, you really need
0:24
to have a front end. I mean, all you have, all you
0:27
need to have is an API or an MCP servers that these agents can
0:30
interact with and then provide the capabilities that.
0:39
I guess we both are kind of on vacation right now.
0:41
I'm in Portugal, you're in Italy.
0:44
Not bad places to be, man. No.
0:46
Well, I'm Italian, so I spent two or three months a year here
0:49
so when I work my family can enjoy a beautiful Italian
0:52
summer. Yeah, get the Gelato and the
0:56
pizza. Pizza is the Gelato.
0:59
So good. I, I was just in Lugano a few
1:02
weeks ago, which it isn't, it isn't Italy, but it kind of is,
1:07
you know? It's not too far.
1:09
It's not. It's like I I'm from Milano
1:11
originally. Actually my father is from here,
1:13
but I grew up in Milano, so I used to go to Lugano all the
1:15
time. It's like an hour away.
1:16
Nice, nice. Well, I mean, you've been doing
1:20
stuff in the tech field for ages.
1:25
I think what people probably know you for these days is being
1:28
the CTO of Kong, which is a, in my mind, a very successful
1:32
company. You all have been doing some
1:34
incredible things. Maybe for the uninitiated, can
1:38
you give us the TLDR of what Kong is?
1:42
Kong is a company that provides a software for managing
1:45
connectivity, API connectivity, micro services connectivity,
1:49
ingress controller connectivity. And of course now that AI is
1:53
becoming one of the major drivers of API connectivity in
1:57
the world, also AI infrastructure.
2:00
So we usually run with developers.
2:03
We work with developers with organizations that have AP is or
2:06
wants to build AP is and then need infra to be able to secure
2:10
it, manage it, and so on. So how did you start to take
2:14
notice of all of the agentic workloads and almost, I imagine
2:20
you probably maybe had some sleepless nights when you're
2:23
thinking about, OK, if agents are the future, what does that
2:28
mean for AP is? And as you said, your vision is
2:32
much bigger than just like, oh, we're not an API company, we're
2:35
a connectivity company. So what's the next step there?
2:38
Can you talk me through your reasoning process?
2:43
Yeah, for sure. No, for sleepless nights, that's
2:45
the right way to put it. You know, AI is one of those
2:47
things that happens once in a while, once every half a decade,
2:50
the once a decade that fundamentally changes how you're
2:53
thinking about the world, how you're thinking about customers
2:56
or users using your products in this new world.
3:01
And of course, for us has been a big driver of API traffic.
3:04
So Kong being very much involved in the API space when AI became
3:09
popular, you know, all the models, the LMS, CHA, GPT, you
3:12
know, and Tropic and so on. Well, it turns out that all of
3:16
these LMS were powered by APIs. And so as you were trying to
3:19
build applications that harnessed the intelligence of
3:23
all of these models, then you had to create more API traffic
3:27
to be able to to consume it. And then these new agentic world
3:31
where basically agents are now becoming these units of
3:36
intelligence or execution that can do things for you.
3:39
They use models to get that intelligence.
3:43
But in order for an agent to be capable, it also needs to
3:46
integrate with an ecosystem of API is so that it can be capable
3:51
to do something useful and smart.
3:53
Because you know, our agents if they don't have access to AP is
3:57
they could use the best model in the world, but they are not
4:00
going to be able much to do much with with with that model if
4:04
they cannot interact with the real world data and services.
4:07
And typically these things are offered via an API.
4:10
Now of course, now we're seeing in industry that traditional
4:12
Restful AP is in the agentic world are now being a little bit
4:16
replaced with new protocols, NCP being one of them.
4:20
But the point being is that it's just a new API that we're now
4:23
leveraging and using to build this new type of application.
4:26
So all of this was happening in front of our eyes and and of
4:30
course being very much involved into this space, how, how could
4:33
we not see it happening in developing?
4:36
Yeah, I can imagine it's something that you're probably
4:39
keeping tabs on and then all at once it was like, well, we need
4:46
to be very involved in this. If things do shake down the way
4:50
that people are expecting them to shake out and a genctic
4:55
workloads become more and more predominant, then that's just
4:59
like completely white space for you to go and capture some of
5:03
the market. But also from the tech
5:06
standpoint, it's probably really cool to think about how does
5:10
what we were doing with AP is, how can that translate over to
5:15
MCPS and how can we bring a bit more maturity to the MCP world.
5:21
And I imagine you have been doing stuff with like working
5:26
groups in the MCP community and that type of thing.
5:31
Have you recognized like where folks are enjoying using MCPS
5:38
and where folks are not? Yeah.
5:42
Well, I think MCP, everybody enjoys enjoys having that MCP
5:46
server up and running and then being able to use it just
5:48
because the ecosystem is now integrating with it.
5:50
So you know, the, you know, MCP clients are now in the our IDs
5:55
that are in the, you know, the the prompts that we typically
5:59
use to ask questions to AI. So I think that once you have
6:01
it, it's actually quite enjoyable to use.
6:04
The problem is building it. There is lots of organizations
6:08
out there that are basically hiring developers to go build
6:11
MCP wrappers on top of every API.
6:14
You're right. You know, once we saw these AI
6:16
wave coming in, given that we consider AI as being the latest
6:21
driver for API adoption. The more AI, the more API.
6:25
You know, we also built infrastructure products that
6:27
help with MCP by the way. So it's not just an AI gateway
6:30
for and Lens that we provide, but also an AI gateway for MCP.
6:34
And we also have introduced the ability to automatically auto
6:39
generate MCP servers from any Restful API that you may already
6:43
have. So the goal truly is to remove
6:46
the unpleasant part of MCP, which is building it, scaling
6:49
it, deploying it, putting it behind a load balancer.
6:53
You know, we can get rid of all of that as long as you have an
6:55
open API spec, just take that spec and we generate an MCP
6:59
server on the fly. But then once you have these
7:01
ecosystem of MCP servers, it becomes quite nice because then
7:05
you can now build a catalog of MCP servers that your agents are
7:09
going to be able to use. Now the problem with MCP is
7:13
security. I mean, there I think there is
7:14
still lots of open questions on how we are going to be dealing
7:18
with security, especially when we're trying to impersonate the
7:21
end user when consuming an MCP server.
7:24
I think that we as an industry, you know, all together have to
7:27
come up with a solution for that.
7:30
But I'm sure that, you know, the, the specification is moving
7:33
very fast. It's evolving very fast.
7:35
They just added security based on all out, you know, at the
7:39
front end of the MCP server. So we will have also to figure
7:42
out how to implement the authorization process across all
7:46
the MCP servers that you can consume.
7:48
But, you know, once we figure that out, I think it will become
7:50
even more powerful and even more widespread because then end
7:54
users are going to be able to use an MCP server that can
7:57
impersonate, let's say their Gmail account, their, you know,
8:03
Google Docs account there, you know, Uber API account and so
8:08
on, and then start doing more interesting things with it.
8:10
Exactly as soon as my agent can understand that I bought tickets
8:16
to the Dead and Company show and I need an Uber there and
8:21
definitely back then I will be happy.
8:25
I bet I don't need to organize any of that it it gets done.
8:29
Or even I'd like to reference like the date night example
8:33
where I have an agent that understands I bought tickets to
8:37
whatever, like the orchestra and there's two tickets and I'm
8:41
going with my wife and so it can organize a dinner, maybe get
8:46
reservations, all of these things that would be very nice
8:50
to offload to an agent. Right now.
8:53
It feels like what you were saying is it's really cool for
8:57
engineers to stand up these MCP servers and folks like the idea
9:03
of them, but actually putting them to use and getting a lot of
9:08
mileage out of them. In my experience, I haven't seen
9:12
as much, but maybe you've been in a different field because you
9:16
were so far into it. Are there cool use cases that
9:20
you've seen with MCP servers that are in production?
9:23
They're being used by and they're being consumed by a
9:27
large amount of folks. I think, I think it's starting
9:31
now. I mean, organizations, you know,
9:32
this is moving so fast, it's moving faster than the
9:35
organizations are able to adapt. But, but there are very smart
9:39
progressive teams that are AI native inside of the, you know,
9:44
organizations that are dealing with our health insurances,
9:46
banking, you know, the kind of type of customers that we work
9:49
with, you know, digital and technology companies as well
9:53
that are starting to harness and use MCP more and more.
9:56
Now The thing is, you know, there is still not much
9:59
infrastructure available for running MCP at scale.
10:02
So what many of these organizations have been doing,
10:05
it's also kind of reinventing the wheel and building NCP
10:08
servers, but then also building the NCP analytics that MCP
10:12
security, the MCP catalogs and so on and so forth to be able to
10:16
make MCP usable and governable within the organization.
10:20
So part of my job as ACTO is to observe the pattern of people
10:25
duplicating their work and try to find a way to bring all of
10:28
that duplication inside of our products so that our customers
10:31
can focus on building the best agents the world has ever seen.
10:35
Instead of having to build all of these cross cutting
10:38
requirements that are needed at the infrastructure layer to be
10:41
able to then run at scale. Right.
10:43
So that's that's part of of, you know what, what I'm trying to
10:46
notice. Yeah, that's such a good point.
10:49
Like where are the anti patterns so that you can help and then
10:53
ideally throw that in the product and then people don't
10:56
have to think about it because it's in it's been abstracted
10:59
away for them. No, for sure.
11:01
By the way, you mentioned something that's very
11:02
interesting. You know, in your example of
11:04
going to concerts, buying tickets, ordering food,
11:07
something that's clear that may happen.
11:12
There's a very high chance that it's going to happen in the
11:14
future is that the way we're using third party software or
11:20
APIs or services is going to fundamentally change.
11:22
I do believe and I truly believe that the agents or even the
11:26
prompts is the new browser. You know, we're not going to go
11:29
on a browser, click around, you know, do things like we were
11:32
doing in 1999 or 2022, but we're going to be leveraging agents to
11:37
do the job for us. And when there is an agent on
11:39
the other side, you really need to have a front end.
11:42
I mean, all you have, all you need to have is an API or an MCP
11:45
servers that these agents can interact with and then provide
11:49
the capabilities that that you want.
11:51
And so, you know, in a world that's dominated by AI, where
11:53
agents are becoming more and more an extension of ourselves
11:57
to be able to do some of the things that today we do
11:59
ourselves, but the agents will take over and do for us.
12:02
I think that that, you know, concept of front end perhaps is
12:08
also going to be transformed and revolutionized.
12:11
I mean, is the next Airbnb going to be a front end when people
12:15
can click and you know, like today, or is it going to be a
12:18
list of curated MCP servers that are going to be showing you
12:21
previous of rooms, previous of prices and all of that in a
12:24
prompt via a prompt, right? So I think I think that we're
12:28
just at the beginning of a massive revolution on how we as
12:32
humans are using technology in our daily lives.
12:36
Yeah, the interface with the model being like the new browser
12:41
is something that I heard my buddy Simba talking about last
12:45
time I was in San Francisco. And I also remember right when
12:51
ChatGPT was getting popular back in 2022, my other buddy William
12:58
was saying, you know, man, it's just going to be agents to
13:01
agents. There's not going to be like me
13:04
scheduling anything with you. We're just going to let our
13:07
agents do that. And you've kind of seen that
13:09
happen a little bit because that's a very, I guess you could
13:14
say like that's a very understandable example.
13:17
But I wonder if back in the day when the Internet was getting
13:22
popular, if there was so much talk about what is possible when
13:27
we connect all of these different computers and when we
13:31
have this open communication with people all around the world
13:34
instantaneously. And I remember that like 90% of
13:40
what we thought it was going to be was completely wrong.
13:44
However, I wasn't actually like deep into the tech world when
13:48
that was happening. So I don't know too well if
13:52
there were folks that were like, yeah, they were the
13:55
Nostradamus's and they could predict what was going to happen
13:58
with the Internet. But all that is to say like
14:03
we're understanding the agent world through our current
14:08
context of the way that we interact with the computer.
14:13
And I'm, I'm constantly asking myself like, how can I think
14:19
bigger to understand what is possible if we do have the
14:24
agents 24/7 going on? But this change, if you think
14:28
about it, has already happened in the past.
14:31
What did we have before the Internet?
14:32
We had Yellow Pages. So what was the main way for a
14:36
customer to get in touch with a business?
14:37
It was the fault. You would take your Yellow
14:40
Pages, find the business and call, pick up your phone and
14:42
that's, and that's how the businesses would transact.
14:45
Then the Internet came, the browser came, and now all of
14:49
those businesses that used to have, you know, these, you know,
14:52
entries in the Yellow Pages, they all started to create
14:54
websites. Why?
14:55
Because now websites becomes the main way that people start
14:59
interacting with those businesses.
15:00
The businesses will always change and adapt to provide the
15:03
right mean of communication so they can still communicate with
15:06
their customers, capture customers or, you know, and, and
15:09
earn their business. And therefore, if customers stop
15:13
using the browser more and more and they start using agents more
15:15
and more to drive their business.
15:19
Likewise, in the past, the Yellow Pages were replaced by
15:22
the browser. The browser will be replaced by
15:23
the prompt. I mean, it has happened before
15:26
and it will inevitably happen again if customers start
15:29
interacting with these businesses in a different way.
15:33
And the businesses that do not adapt that those are the ones
15:35
that will be replaced by the ones that do.
15:37
Imagine being a business when the Internet becomes popular
15:40
that doesn't, you know, refuses to create a website and still
15:43
insists on just being on the Yellow Pages.
15:45
You know, that business, that's it.
15:47
You know, there's there's not any, any future whatsoever for
15:50
that business, you know, and they're going to be replaced by
15:52
the ones that do cater to these new type of outings.
15:55
Actually, you bring up a great point, which is almost that
15:58
Bezos ISM of let's focus on what doesn't change as opposed to
16:03
what is potentially going to change and how it's going to
16:06
change. And you're, you nailed it on the
16:10
head when you're like, businesses are still going to
16:13
need customers. They're still going to need to
16:15
interact and talk with customers in some way, shape or form and
16:18
get the customers to know about what they do and how they do it.
16:23
And maybe there's promotions. There's all that stuff
16:25
happening. The shape of how that looks,
16:29
whether it is in the Yellow Pages or it's through the
16:31
browser now it's through agents is the thing that will be
16:36
changing. Not necessarily like the higher
16:38
level. I'm a business.
16:40
I need to make money. The way I make money is through
16:42
clients. So let me get clients.
16:46
They'll just go wherever the customers are.
16:48
And, and if the customer becomes the agent, because you know,
16:51
when once we have an army of agents taking care of our lives,
16:55
taking care of our bookings, taking care of our reservations,
16:57
taking care of everything, then effectively the agent almost
17:00
becomes the customer to all of these different services.
17:03
And so the businesses that will thrive in an agent agentic world
17:07
are the ones that can cater better to these new type of
17:10
customers, the agents themselves.
17:13
That actually brings up a thought of we have SEO right now
17:19
for when we're searching for companies and we need something.
17:25
But there's not like the MCPSEO type of thing where you have the
17:32
model recommendations where if you're dealing with the
17:37
foundational model, if you are recommended in the cloud output
17:42
or the ChatGPT output, that's great.
17:44
But I I also wonder if the agents are going to have
17:49
preferences. Maybe it's that certain MCPS are
17:53
better written or the servers are nicer.
17:56
So you almost get like this SEO from that.
17:59
I know this is completely tin hat hat type shit but it's fun
18:05
to think about. Like Wow, could we have specific
18:10
SEO at the MCP level as opposed to the model level?
18:14
Maybe it's all dealing with the model because the model is what
18:17
is the brain at the end of the day.
18:21
You know, it's a very provocative thought.
18:23
I think that, you know, marketplace dominated by humans,
18:26
you know, our modern advertising, you know, works,
18:30
you know, the louder, the brighter and you know, and
18:33
that's how you win your the business.
18:35
Who knows, perhaps the marketing of in an agentic world, it's not
18:38
being the louder and you know, the brightest or the biggest,
18:40
you know, billboard in the world anymore.
18:42
It is having the lowest latency. So you want to earn more agentic
18:47
business then you have to provide the lowest latency
18:49
because it means that the agents are going to be completing their
18:52
job in a quicker time frame. Yeah, dude, that's.
18:56
That's fun to think about. And that just comes back to
18:59
like, why it's, it's so interesting the spot that you
19:03
sit at, because I, I imagine that's something that you're
19:06
thinking about like, oh, how can we make these, whether it's MCP
19:10
servers or it's just AP is the best that they can be so that A,
19:16
people don't have to have headaches with them, but B, the
19:20
interactions with humans or agents can be the most optimized
19:26
1. 100% you see, I think the technology is most likely will
19:32
change. I mean the underlying
19:34
technologies that are going to be powering the agentic world,
19:37
perhaps they will change and perhaps LMS as we know them
19:40
today and they will evolve into something else.
19:42
You know MCP as we know it today will evolve in.
19:45
But the idea that there is going to be something that can make
19:48
decisions for us. I think that that idea how, you
19:50
know, once it's out of the of the bottle, you know, the genies
19:53
out of the bottle, then you know, it's not going to go back
19:55
in. And I think that that will
19:56
happen. Now the question becomes, what
19:59
are going to be the technologies that are going to be enabling
20:01
this world? And you know, you work with the
20:03
community, you work with ecosystems, with the foundations
20:06
to make sure that you, you know, you understand what what that is
20:09
going to look like. But there is no doubt.
20:11
That the world we knew from the early 2000s until you know,
20:16
until now, it's going to be the same world.
20:18
It's going to dominate in the next 10 to 15 years.
20:20
So this is happening. It's happening whether we like
20:22
it or not. And my job as ACTO is to ensure
20:26
that if someone is out there building infrastructure for this
20:30
new world, while it better become instead of somebody else,
20:33
so that, you know, we can, we can capture these early
20:36
adopters, we can support them early on with their
20:38
infrastructure work. And we're doing it today, by the
20:41
way, you know, our AI gateway is rapidly growing, you know, from
20:44
a business standpoint, because every organization in America
20:47
right now in the world, not just in US, truly is now diving in
20:51
and jumping into these agentic opportunity because they do
20:54
understand that if they don't do that, somebody else will and can
20:58
steal the opportunity from them. You see companies are made or or
21:04
or disrupted during these massive periods of transition.
21:08
When the mobile comes in massive, the Internet first and
21:11
foremost, then mobile comes in. You know, if you are not on
21:14
mobile, you're dead. You know, you have to have a
21:16
mobile strategy with a mobile application.
21:18
Then it's the cloud massive transformation and then now it
21:22
is AI companies, new companies emerge and you know, older
21:28
companies fail when they see these new waves of
21:32
transformation coming at them, but they avoid to disrupt
21:37
themselves to be able to capture these opportunities.
21:40
And that creates a window of opportunity for for somebody
21:43
else to jump in and steal the opportunity from them.
21:45
We've seen that, you know, in the past with Blockbuster and
21:47
Netflix are your typical case study of organizations that are
21:51
not adapting or you know, Kodak when we transition from analog
21:55
cameras to digital cameras. And today we're transitioning
21:59
from a pre AI world into an AI world.
22:03
And likewise every other transformation before companies
22:06
that see this opportunity are the ones that are going to win.
22:10
And so we're seeing lots of organizations right now that
22:12
understand that the threat and strategic risk, but also the
22:15
opportunity of investing early on in AI because this is what
22:18
where the world is going. And they are also partnering
22:20
with Kong and with us and others to provide these early
22:24
beginnings of an agentic world first inside of their
22:29
organization to transform their business processes.
22:32
But some of them are also now offering these as new products
22:36
that the customers can use to increase the retention, to
22:39
increase their engagement, to cross sell more products.
22:41
So we're starting to see the beginning of AI being used
22:45
internally, but also externally. And I think it's very exciting
22:48
because right now it is a very creative space.
22:51
Everybody's trying everything and see what works, to see what
22:54
doesn't. And it's just very exciting to
22:55
see, actually. Along those lines, what are some
22:59
of the coolest use cases or most creative, most awe inspiring use
23:06
cases that you've seen? Well, so first.
23:08
And foremost that marketing departments around the world are
23:10
being revolutionized. I mean the ability to, for
23:13
example, articulate to the value of products, being able to
23:16
enable a sales force, you know, all of that is being automated
23:19
with AI. I've seen also AI powered sales
23:22
teams now, by the way, that's another angle.
23:25
You know, today we're selling to humans.
23:26
Maybe in, in in the future we'll be selling to agents.
23:29
How do you even communicate and advertise to them?
23:31
You know, going back to the conversation we had earlier, are
23:34
we going to have sales teams or are we going to have agents
23:36
selling to other agents? What does that going to look
23:38
like? So I'm seeing some of that.
23:41
I'm seeing lots of innovation in the video AI space, the ability
23:45
to take an image of me holding a cup of coffee and then
23:49
leveraging AI to creatively replace that cup of coffee with
23:53
a bird, for example. And you see like a bird just
23:55
flying on my hand instead of instead of, you know, your
23:58
original picture that I had therefore the ability to create
24:02
small movies just by thinking about what I would like to see
24:06
and what I would like, you know, AI to come up with.
24:08
So I think there's going to be massive disruptions in how we
24:10
generate content. We've seen that already, massive
24:13
disruptions in creative spaces like video Hollywood.
24:16
You know, why should we wait for Christmas for to watch our
24:19
favorite movie when we can invent a new movie every day?
24:23
And AI is going to do that for us as well.
24:27
There's going to be lots of disruptions in in the legal
24:29
field. And you know, there, it's all
24:31
over. It is permeating every area of
24:34
society. That is what makes it also quite
24:37
exciting because we are just at the beginning of a massive
24:40
transformation in the world. And this is when good companies
24:43
emerge, the ones that can take advantage of this opportunity,
24:46
they emerge to be able to care to these new world.
24:48
So we're going to be seeing lots of transformation.
24:50
And whenever there is transformation, there is
24:52
excitement, at least from my part, because I like to see what
24:55
everybody's doing and where everybody's ending up with.
24:57
You know, financial services companies are now creating AI
25:00
powered wealth advisors, Wealth advisors that used to be only
25:05
given to ultra high network individuals, while they're now
25:08
creating AI powered wealth advisors for portfolio analysis,
25:12
investment strategies and so on that everybody can use.
25:15
This is creating a good positive effects in their business
25:20
because now everybody has more engagement, everybody invests
25:23
more, everybody is, is basically leveraging their services in a
25:27
much better way. So AI is being used already in
25:30
some of these use cases and this is just we're just scratching
25:33
the surface. So now getting a little.
25:36
Bit into the weeds of the AI gateway that you were talking
25:39
about. Are you seeing that folks are
25:45
just hitting rate limits left and right?
25:49
Is it like, well, it's the impotence behind why you need a
25:53
gateway and the value it brings? Is it just switching certain
25:59
tasks to local models or? Very safe.
26:04
Models and then other tasks can go to open AI or clawed, that
26:08
type of thing. Like how what's the shape of
26:12
these gateways? I I know it's hard to say a
26:16
general blanket statement, but maybe you've seen some patterns
26:19
that you can call out. Well, the AI Gateway provides
26:24
infrastructure for AI agents. That means it provides
26:27
infrastructure to consume the lands and infrastructure to
26:30
consume the MCP servers and APIs.
26:32
Because we need both of these things to create capable agents.
26:36
So what it provides is a whole layer of governance to be able
26:40
to expose what the APIs, what models we want our developer
26:45
teams to be able to use when they're building agents and
26:48
being able to govern that. So tiers of access to the
26:50
models, tiers of access to the MCP servers, the ability to how
26:53
to generate MCP servers from every API, the ability to
26:56
implement the security, authentication, authorization
26:59
and all of that. Then there is a whole layer of
27:01
observability that's being extracted out-of-the-box.
27:04
You know, whether we have one agent or a million agents
27:07
running across multiple teams, we need to have a an
27:10
understanding of what is that these agents are doing, what
27:14
models they're using, how many tokens, what you know, what's
27:16
their latency, what APIs they're using.
27:18
So all of these observability, we're talking about hundreds of
27:21
metrics can be easily collected using the AI gateway technology.
27:26
Without the AI Gateway, we would have to build security all the
27:29
time inside every agent, would have to extract observability
27:33
all the time inside of every agent.
27:34
This basically removes all these cross cutting requirements away
27:38
from the developer teams that can focus now on doing what they
27:41
should be doing and that is building the best agents the
27:43
world has ever seen. So the time to market gets
27:45
quicker while at the same time we have control and governance
27:48
on top of that. But then it also helps with the
27:51
choosing different models. So for example, Cong as a a, we
27:56
created a universal API where you can build your agent once
28:00
and then the flip of a switch you can transition to different
28:03
models even across different vendors.
28:05
So you can build once and then go from open AI to Azure, from
28:09
Azure to Bedrock, from Bedrock to data bricks without having to
28:13
rewrite the agent code. So it's a universal interface
28:17
that allows you to choose the best model for the job to load
28:20
balance and route across the best model for the job.
28:23
For example, we may want to route very simple prompts to
28:26
cheaper models, and we wanted to route more complex prompts to
28:30
more expensive reasoning models. And we can then more efficiently
28:35
manage our spend and we can more efficiently manage the latency
28:38
of those responses. Things like a guardrails, things
28:41
like semantic acceleration of our requests.
28:44
Think of a building, a semantic CDN for all of the AI responses,
28:48
the ability to implement rag pipelines in a much better way,
28:50
the ability to sanitize PII information without having to
28:55
ask developers to do it in every time they build an agent.
28:58
So we can apply blanket rules to sanitize all PII Social Security
29:03
numbers, name addresses, you know, all the PII data in such a
29:06
way the organization has a higher comfort level that these
29:11
data, it is not going to be leaked.
29:13
You know, models are notoriously very good.
29:17
You know, they're, they're very bad at keeping secrets or, and
29:23
they'll, and they, they like to disclose all of this information
29:26
or if asked the right way. And so being able to provide a
29:30
mechanism to enforce the security.
29:31
I mean, we need all of these things to go in production with
29:34
AI. It's unimaginable for an
29:37
organization to build agents that others are using or even
29:40
internally without having a certain degree of comfort on how
29:43
they're being governed, used, observed, managed, secure.
29:47
I mean, it's just the 1.1 a of building agents, yeah.
29:52
It's very much table stakes and that's why I think a lot of
29:57
folks are still in the phase of we're testing, we're doing the
30:03
POC's, we're trying to get it to work.
30:05
But until you get all of that enterprisey type of stuff down
30:12
and clear and at least have a story and have checked all the
30:17
boxes, it's very hard to just be like, let's yellow this into
30:21
production and see what happens. No, I mean, it's like I said,
30:25
it's table sticks and you know, developers also, these may not
30:29
necessarily be their area of expertise.
30:32
And so you know, we want, we want to enable as many
30:34
developers as possible in the organization to be able to
30:37
innovate and with AI and build agents and so on.
30:41
But we cannot do that if whenever they start, we're
30:45
asking them to do 100 million things that goes beyond creating
30:49
the actual agent. So I truly believe that this
30:51
creates and by the way, platform teams exist for a reason.
30:54
And that is they existed to identify these things that
30:57
everybody needs to do, take them away and centralized them in a
31:01
core platform, in a core service that then can be given to
31:04
different lines of businesses, different developers and so on.
31:07
So on one-on-one end, we work with the platform teams to
31:10
provide these infrastructure they, they can control.
31:13
But then on the other hand, we're also accelerating the
31:16
developer teams that are building agents because now they
31:18
don't have to do all these other stuff.
31:20
So it's much easier for them to kick, you know, to, to, to, you
31:25
know, hit the road with their own agents and, and get started
31:29
now. How are you going about making
31:33
the decision on where and what to invest in?
31:37
Because especially in the AI space, there's a lot of noise
31:41
out there and a lot of different product directions I imagine you
31:45
could take. Maybe with MCP it was fairly
31:48
obvious. So you thought, OK, we're going
31:51
down that road because it just has so many similarities and
31:54
parallels to the API world, but with other areas.
31:59
Like I'm guessing the AI Gateway wasn't your first foray into
32:05
creating AI products. And so maybe you have some
32:09
products that died on the cutting board or maybe you tried
32:12
to put them out and you just didn't see the adoption.
32:14
Like you were saying on that playbook.
32:15
Once you see a certain amount of ARR or certain type of adoption,
32:19
then you can graduate it to the rest of the customer base.
32:22
Were there ones that you tried that did not get that type of
32:26
adoption? You know, I got to say something
32:29
when when I decide to start something are usually pretty
32:33
high convention, you know, because why, why do something if
32:36
you're not highly convinced that it's the right thing to do.
32:41
And so, you know, from the office of the CTO, we, you know,
32:45
at Cong, we created the first Inglis controller, you know, to
32:47
target Kubernetes workloads. Kubernetes became, you know,
32:50
ubiquitous, kind of popular. Yeah, so that worked.
32:53
When service mesh became a industry trend, you know, we
32:57
came up with a very compelling, very powerful service mesh
33:00
product that went very well. You know, I think the service
33:03
mesh market right now, you know, AI stole the Thunder.
33:08
Let's put it from, you know, service mesh in I, I would, I
33:10
would, you know, I'm, I'm seeing, but you know, it was a
33:13
very successful product that grew and was transferred to the
33:16
R&D organization. Then insomnia was another
33:20
product today at the office of the CTO to cover and we were
33:22
able to 10X our revenue in, in less than 18 months.
33:27
So we also were successful with that, you know, with AI now we
33:31
did the AI gateway that also has been very successful.
33:34
So I'm not saying that I have, I have not done mistakes, but The
33:37
thing is if you start with something and you keep iterating
33:40
with customer feedback and you keep micro pivoting, you never
33:45
really fail because you are basically simply realigning your
33:49
product to where the market is going and where the customers
33:52
are going. So I would say that there hasn't
33:57
been a project right now that I've cancelled simply because of
34:00
the high convention I have when I start something.
34:03
So it's more along the. Lines of the product you first
34:07
created in the beginning was a vastly different product than
34:12
what it is today because of all those micro pivots actually.
34:15
Yes, you know what, I'll give a product that we shut down and
34:19
that was the original product. You're absolutely right.
34:22
Kong is the result of a pivot. Our first idea was an API
34:27
marketplace because, you know, back in 2010 when we started
34:30
this business, we were thinking if the world is going to be
34:32
dominated by APIs, which ended up being true, then the world
34:37
would need an Amazon of AP is where sellers and buyers can
34:40
transact AP is. And so we wanted to build out
34:42
the first Amazon for AP is or marketplace of AP is.
34:46
Well, in 2010 people were asking us what is an API?
34:48
You know, nobody had any any clue.
34:50
But then obviously eventually AP is became real.
34:54
So that was the good bet that we made.
34:56
The bad bet that we made was that the marketplace was still
35:00
early on and it didn't really, it really didn't really work O
35:05
when we decided to build Kong Kong, actually it was the
35:09
technology that the marketplace was using internally to process
35:13
all of these API requests. And so we took the most valuable
35:16
thing that we had built for ourselves, our technology and we
35:19
extracted it and then offered it stand alone.
35:22
And that's how Kong came to life.
35:24
And then the marketplace was shut down and sold to another
35:27
company. So we did, we did shut down mash
35:29
ship. I don't consider Mash Ship to be
35:31
necessarily one of those, Oh, you build something, you shut it
35:33
down because in reality, thanks to the marketplace, Mash Ship
35:37
was called. Thanks to the marketplace, we
35:39
were able to then build Kong and sort of the spirit of mass shape
35:44
continues to live to this day because of Kong which which was
35:49
the the biggest product market fit we found back in those days,
35:52
no. Talk me through how you get the
35:55
conviction. Like what is?
35:58
Something that. Will get you enough conviction
36:03
to say we need to go and build around this and then you figure
36:07
out the product you do those like I really like that the
36:10
course corrections the mini course corrections so that
36:13
you're always aligning to what the customer's asking for but.
36:18
What is it? Before that, that you, you need
36:21
to see to be like, yeah, we're going for this, you know?
36:26
I think that at this point I'm not looking at being necessarily
36:32
the first ever to do something. I may actually look for early
36:37
indicators of somebody else doing well into an emerging
36:40
category and then, you know, try to find out ways on how we can
36:46
potentially even go even further and even more advanced in that
36:48
category. And you know, I think that being
36:52
first ever, I mean away Kong API Gateway was the first ever open
36:59
gateway in the world. You know, I know how hard it is
37:01
to be first in, in absolute terms.
37:03
And it's very unlikely for every founder or organization to be
37:08
the first ever in a new field is just very unlikely.
37:12
It took us six years to to be first with Kong Gateway and six
37:18
years of trying, working 24/7, sleeping in the office, sleeping
37:22
on a couch. I know how hard it is.
37:24
So now that Kong as an organization grew a little bit
37:26
more, you know, we have the luxury preps to wait a little
37:30
bit longer. And and by that I don't mean,
37:32
you know, it's still, we're waiting six months, right.
37:34
So it's it's not like we're waiting for five years.
37:36
We're waiting for six months just to see that something
37:39
that's beginning to get traction in the industry works well.
37:41
And at that point, we decided to go in with very high conviction.
37:46
What we do at the end of the day ends up being always a good
37:49
choice, a good product for the users, for the customers,
37:52
because none of the things that we did otherwise would be
37:54
successful if we did not reinvent and reimagine how those
38:00
things would work in the 1st place.
38:01
And so at the end of the day, everybody benefits from
38:04
competition. We benefit, the customers
38:06
benefit and, and the industry benefits as a whole.
38:10
But that's, that's one way that you can reduce that risk because
38:13
the risk it's very high very early on and then it kind of
38:16
drops. It's like the depreciation.
38:19
If you looked at the risk of, of starting a new business and you
38:21
looked at the depreciation chart of a car they brought, of a car,
38:24
it would probably look the same. You know the first, the first,
38:27
the first, you know 5% of the of the life of a car, the
38:31
depreciation just hits all at once.
38:33
And and likewise with risk when doing something new, the biggest
38:37
risk is the beginning. If you wait only, you know, 4-5,
38:39
six months, you're already materially reducing that risk
38:43
and you can go in with still a very high risk level, but not as
38:46
higher as it would have been if you did it from scratch.
38:50
So you. Have the office of the CTO, is
38:53
that a group of is that like a small team that you've curated
38:56
that are able to do this almost like R&D with you?
39:00
And what is the composition of that team look like?
39:04
Is it a lot of visionaries or is it just builders, executors, a
39:08
little bit of both it's. Builders, people that have
39:12
passion in building. Whenever I need to build
39:16
something new, I actually rebuild my team from scratch all
39:19
the time. So I really, I really like
39:20
hiring and I really like building as well.
39:25
And so when we build something and then it works, we transfer
39:29
the whole squad, the whole team to the main R&D organization in
39:35
such, in such a way that they don't disrupt the products
39:38
themselves. Because imagine, you know,
39:39
having a group of people building products, then you
39:41
transfer the product you still want to the people that have
39:43
built that product to still continue working on it.
39:46
We transfer that whole thing. That leaves me with a big
39:49
problem that every time I want to start something from scratch,
39:52
I need to basically go and rebuild a team.
39:54
And that's great because sometimes they go build things
39:56
in different directions. And the team I've hired to build
39:59
an infrastructure product might not be the same team that I'm
40:01
going to be hiring if I'm building a developer and user
40:04
product. And so I go ahead, leverage my
40:07
network, hire very quickly and go and build and hit the road
40:11
with, you know, a new idea, a new product, a new team and try
40:15
to ship usually within six months.
40:17
So we we work in a very relentless way, lots of energy.
40:20
And I think that's what happens when you really believe into
40:22
something and you have high convention into something that
40:25
you just don't waste time. You just go keep going and keep
40:27
running since they want. And it turns out that, you know,
40:32
the teams love it too, by the way, because they get to work on
40:36
basically a startup within the startup.
40:38
So they get to experience that, you know, they have lots of
40:41
ownership, lots of decisions that they can make and I want
40:46
them to make because of absolutely, you know, the first
40:48
time people building a product should be absolute owners of the
40:51
product. So we we listen to all ideas, we
40:54
listen to all decisions. We go ahead, we build.
40:56
I bring them in front of customers, I bring them in front
40:58
of the users. And then there is nothing better
41:01
than looking back and be like, oh, you know, wow, this thing
41:04
look like a crazy idea 12 months ago, 18 months ago.
41:07
And now we have 200 evaluations ongoing when perhaps you know,
41:12
20 or 30 customers that are currently using the product in
41:15
production to transform how they're doing things in their
41:18
own organizations. So I think it's very rewarding,
41:20
but it's for sure very high intensity.
41:24
And to be fair, I don't know any other way to build new products
41:27
other than having a high energy, high intensity and and also lots
41:31
of purpose. Yeah, I'm.
41:33
Almost like jealous of folks in your position that have a strong
41:38
customer base that you can go and you can talk to at any point
41:42
in time. Because I know how hard it is
41:46
for founders to try and find that product market fit.
41:51
And being able to talk to existing customers feels like
41:56
you get to almost short circuit that loop and get into those
42:01
conversations faster, easier. And folks are already trusting
42:06
you with one product. So they're probably going to be
42:08
very forthcoming when you're asking them different ideas,
42:13
people. And companies have a reputation.
42:15
So our ability to innovate and deliver in the past certainly is
42:19
a lower risk in the eyes of an early adopter for a new product.
42:24
So I mean, for sure, I think, I think it does help, but you
42:29
know, I, it does help to get to the first input perhaps, but
42:33
then to continue working with the user, working with a
42:36
customer and continue building that product, you know, that's
42:39
back on, back on the shoulders of the team, you know, So I, I,
42:44
I think it does help. Probably it does not help as
42:48
much as you think it it would, because at the end of the day,
42:52
when you're building something new, the truth lies in how good
42:55
you're executing and whether this customer was already a
42:59
customer of yours or not. If the product is not good, then
43:02
the product is not good. So it doesn't matter how that
43:04
first input happens, it's not. Like they're going to put up
43:07
with a shitty product if they're already using one of your
43:10
products or not. Look, if you're going if.
43:13
You're going to a restaurant you really like.
43:15
You know, you may try the new appetizer, but if the new
43:17
appetizer is not good, well, you'll tell them it's not good.
43:20
Yeah. Yeah.
43:21
And you're definitely not going to order it again.
43:23
That's and you're not going to order it.
43:24
Again. And do you?
43:26
Normally do it with like design partners, like a startup would
43:29
where you choose 3 or 4 design partners.
43:32
You work very closely with them because I imagine a team of 10.
43:35
You don't have resources to go and talk with half of the Kong
43:39
user base, no. We we usually try to find
43:42
because, you know, also customers exist on a different
43:45
spectrum of willingness to innovate and adopt.
43:49
There are some teams that are very progressive in some
43:51
organizations and sometimes you find these teams in the
43:54
organizations you would expect it built the least, you know,
43:57
like very old school organizations, but very strong
43:59
team. Fine, let's work together.
44:02
They want to transform their business.
44:04
They think that to, to transform their business, they need our
44:07
new technology. Well, the incentives are
44:10
aligned. My job is to make sure that I am
44:12
an extension to their team and build and deliver the best
44:15
technology product they can use. And their job is to adopt it so
44:19
that they can, you know, their business can benefit from
44:23
leveraging these new technology and they're happy with their
44:26
growth. I think you, you want to have a
44:29
few design partners, but I think you also want to be opinionated
44:34
quite a lot, I would say in, in how you are influencing your
44:39
design partners because you know, with a design partner, you
44:42
may also receive all sorts of requests that make no sense.
44:45
And my job as ACTO it is to, it is basically a pattern
44:49
recognition job. I talked to many customers.
44:52
I try to identify what each one of them wants and what it's very
44:55
unique for each customer. And what everybody wants is
44:58
shipping the product. And what's very unique for each
45:01
one of them, I provide an SDK or an extension point where they
45:04
can go and customize it. And it makes them happy.
45:08
And I don't go and and go deep into rabbit holes that I know
45:13
they're not going to create any opportunity for.
45:15
I mean, you know, I know it's it's cliche, but the ability to
45:18
say no is a real thing. You need to say no sometimes.
45:21
Well, yeah. You can step in a lot of shit if
45:25
you just start trying to be everything for everyone.
45:31
Anyway, and I think that's the most reasonable design partners
45:33
are also going to be the ones telling you, hey, you know what,
45:36
we think that what we want is quite reasonable and we would
45:42
like you to implement a solution for this that's also reasonable
45:45
for other customers because we don't want these one offs that
45:48
perhaps in a couple of years are not going to be maintained
45:50
anymore. So the good, the design partners
45:53
are the ones that understand that they're building a long
45:56
term partnership with you or with us and and they want us to
46:01
build a road map that's a good road map.
46:03
It is not Frankenstein all weird.
46:08
Yeah, not their. Road map because they understand
46:10
that if they are the only market for your product, then their
46:13
product is not going to exist for much longer anyways.
46:15
So yeah, unless they're a. Very big client I can imagine,
46:20
but even then you can't have a sustainable product for just one
46:25
client, no? And the good.
46:26
Design part design partners also understand that so they don't
46:29
want to these, you know, big one offs because they know that
46:32
we're we're not going to you know, it doesn't make any sense
46:35
for us to go and build it. So you put into production the.
46:39
AI Gateway and now is there certain things that you're
46:44
starting to see that are starting to peak your interest
46:48
on what's next? Or is it like you're going to
46:51
invest the next one or two years in the AI Gateway?
46:56
How does it work now? Is there like new R&D that
46:59
you're already thinking about all the time?
47:02
All the time, the momentum is very high and it never stops.
47:06
If you, if you looked on my, you know, I have two displays in
47:09
front of me. But if you open up the Jera
47:11
board for AI Gateway doesn't fit.
47:14
So my my job at this point is that to ensure that we
47:17
prioritize properly. You know, it's basically my job
47:20
at this point is the job of APM, right?
47:21
So it is, it is as a CTOI see the opportunity.
47:25
But then as APM, you know, the day-to-day building of a
47:29
product, it's nothing exotic, to be fair.
47:31
You know, it's stand ups plannings, you know, talking to
47:34
the team, talking to customers. So at this point, my job is the
47:36
job of APM and I prioritize my my, you know, there's lots of
47:40
things that we wanted to build. We started with managing LMS. We
47:44
now have realized that MCP is a big requirement and so we rolled
47:50
out one of the most advanced MCP gateway capabilities inside of
47:54
our AI gateway. Even the ability like I said to
47:56
generate MCP servers with from scratch without any line of code
48:00
from any Restful API. And then being able to visualize
48:03
these NCP servers in a catalog that allows developers that are
48:07
building agents to explore all the potential integrations that
48:10
they can use when building it. You know, all of these is very
48:13
innovative. It does not exist out there.
48:15
And so the road map is moving quite fast.
48:19
We're partnering with customers quite fast.
48:21
Just yesterday we closed nine more enterprise customers in one
48:26
day. You know, basically for AI
48:27
Gateway we're talking about organizations that got up
48:30
Fortune 500. So the amount of work is a
48:32
certainly not lacking and there is no, no opportunity for us to
48:37
slow down because if you slow down now, then we lose the, the
48:41
momentum and we lose the opportunity.
48:43
And to be fair, when building a new product, the most exciting
48:46
part is to be in this position. It is to build something that
48:49
finally people want to use. You get lots of inbound, lots of
48:52
requests. And so if, if you have waited,
48:56
you know, for many months for this to happen, for sure, you're
49:00
not slowing down now that you have it.
49:02
So if anything, you grow bigger, you grow faster, you hire more
49:05
people and you try to provide the best value that's possible
49:11
to provide to our customers to, you know, for them to continue
49:16
being customers, for them to expand, of course, and for us to
49:20
be able to then capture and acquire new customers.
49:22
You know, our success is very tight and linked with the
49:26
success of our customers. Because if our infrastructure
49:29
doesn't enable them to do the things that they want to do,
49:31
well, then our infrastructure is wrong, our product is wrong and
49:35
are going to churn and and we're going to lose them as customers.
49:39
And so we're very aligned with all the initiatives that they
49:43
have. And I think that's something
49:44
that's very interesting as being technology vendor like hog is
49:48
that you also get to see how different industries are
49:52
revolutionizing how we interact with them.
49:55
You know, when you one day I speak with a healthcare
49:57
organization, you know, they're revolutionizing how their
50:00
patients, you know, get access to all of their test results and
50:04
so on. You know, that's something that
50:05
millions of people use every day.
50:07
Then the next phone call, it is with a financial services
50:10
organization, you know, they have billions, trillions of
50:14
dollars under management or in their in their bank and they're
50:18
trying to provide new ways for customers to get a sense of
50:21
their financial health, for example.
50:23
And then you speak with another customer, you know, in
50:25
transportation technology, travel, retail, and you are
50:30
seeing the change that AI is doing not only from the eyes of
50:34
founder and CEO of Cong, but also from the eyes of a top
50:38
Fortune 500 healthcare organization, financial
50:40
services, retail, transportation.
50:42
So you're seeing the world changing in front of your eyes
50:45
at 360°. There's nothing that's more
50:48
exciting than that because it gives you so many data points.
50:53
Awesome dude. So Kong is hiring is what I'm
50:56
getting from here.