Latest / Agentic Conversations (formally mlops.community) / Architecting Modern AI Systems: Platforms, Agents, and Integration

Transcript

Architecting Modern AI Systems: Platforms, Agents, and Integration · Agentic Conversations (formally mlops.community)

0:34
All right, we are officially live.
0:36
What is going on everyone, I am excited to have this
0:42
conversation with some incredible people.
0:47
Today we're going to be talking about architecting modern AI
0:50
systems. Let me get right to the point.
0:54
I want to jump in and chat with them.
0:58
So first I will bring out Frederique to the stage.
1:03
Where you at? Hey, there he is.
1:06
How's it going, Sir? Hi.
1:08
Hi, everybody. Let's also bring our next guest
1:13
out here. Shao, where you?
1:15
Hey there. We go how's it going?
1:18
And then last but not least, our final panelist, we've got Alan.
1:25
And so let me set the stage here and then we'll start chewing the
1:29
fat. We really want to talk about
1:31
modatech, Modatech thing, I almost said architecting modern
1:35
AI systems. And to do that, what that comes
1:39
along with, with the platforms, the agents, the integrations.
1:44
So when we explore this specifically, I wanted to jump
1:47
into some of these things that are the most important pieces.
1:51
But before we do, maybe it's cool to hear a little update.
1:55
You all just did a hackathon, I think.
1:58
Who was in the hackathon? Frederick, I know you were
2:00
right. Yes, along with Shao from Bell
2:04
and get help phone tell. Me more about that before we
2:08
jump into the questions and I grill ya.
2:10
Sure. So it was Hackadon we organized
2:13
with Bell Canada and kids help phone around mental health and
2:19
how can we how can we support how how can we build guardrails
2:24
and around conversational agents and making them more secure
2:30
around mental health. So we had over 100 teams
2:34
participating in a hackathon. They were developing their own
2:37
models or building some, some LLM prompts to decide when to
2:44
pass a conversation to a human because it it became sensitive
2:48
or there was issues around suicide ideation and things like
2:51
that. So it was a great collaboration
2:56
and my team specifically was, was how build the evaluation
3:03
pipeline. So we had to evaluate the, the
3:07
submissions from all these teams and we wanted to do it quickly.
3:10
So we build this infrastructure on top of communities running in
3:15
the buzz HPC environment where we had access to LLMS that were
3:21
hosted there, GPUSCPUS. And we, we, we ended up building
3:30
a self-service infrastructure so teams could just trigger jobs
3:35
and see how they fared. And then we displayed those on a
3:38
leaderboard. And it, interestingly, it was
3:42
something we built at the last minute, but it was super used.
3:46
We had 1000 submissions throughout the week.
3:50
It ended up being a game of, you know, where, where do I fare one
3:55
team against the other. So that was super exciting. 1000
3:59
submissions is quite a lot, so Congrats on that success shout.
4:04
You got anything else you want to mention before we jump in?
4:07
Yeah, just starts to write on to it, Frederick said.
4:09
I think sort of I want to highlight like a small
4:11
interesting element element to it.
4:13
Like mid hackathon we actually introduced a new data set that
4:17
almost like sent everybody into a scramble because, you know,
4:20
previously some teams thought they did great.
4:21
And then we have a news for hidden data set shared and then
4:24
people like, you know, we're not scoring as much as we did
4:27
earlier. So I think that's just that
4:29
during a lot of excitement for, for the hackathon, I wanted to
4:32
just take a brief just step back, 'cause I was mental health
4:35
is sort of, I would say very big these days, you know, just given
4:38
the, the sort of crappy economy that we've got, you know, some
4:41
geopolitical tensions. So these companies we've got
4:44
together, we all care about mental health and that we wanted
4:46
to make sure that, you know, we have the right infrastructure in
4:48
Canada to support some of these conversations.
4:52
So I've like, like Frederick's, like a lot, lots of
4:55
participants. And then we had to make sure
4:57
that, you know, we, we built a good support structure to, to
5:01
support our, our, our developers.
5:04
And I think we actually spend a quite a bit of time, you know,
5:06
making sure that when we evaluate some of the
5:08
submissions, they're all, we're, we're looking at both like all
5:11
the presentations, the scores, the documentation, and we'll
5:14
look for innovation and all of that.
5:17
But but yeah, it was a, it was a great time, you know,
5:19
collaborating across these different like companies from
5:22
different diverse background to sort of address a common cause
5:27
of mental health. Excellent.
5:29
Yeah, it is quite an important topic and I think that the idea
5:34
of platforms and what platforms can give you.
5:40
You mentioned Buzz was helping the teams along in this journey
5:46
of the hackathon. I also would love to probably
5:49
kick off this conversation with the idea of where you draw the
5:53
line between what you expect from a platform versus what your
5:59
team needs to own. And then maybe on the next step,
6:02
we can go a little bit further and talk potentially about agent
6:06
harnesses and agents where what you need to own on the agent
6:11
side versus what you would expect someone else to own.
6:16
So Alan, I saw you shaking your head on that one.
6:19
Maybe I'll kick it over to you first.
6:22
Yeah, I can definitely talk about agent harnesses and why
6:25
Buzz is, is a great place for running your LLM model inference
6:31
services to power agents where you know, you're able to avoid
6:34
having somebody looking over your shoulder and you know,
6:37
training on your really great ideas.
6:40
I'll talk about that in a moment, but I do just want to
6:42
kind of talk a quick moment about Buzz and, and what we were
6:46
able to do in the context of that hackathon.
6:49
For those who don't know Buzz, HPC is Canada's largest
6:52
sovereign AI cloud and it's built to give enterprises,
6:55
researchers, governments, builders secure access to like
7:00
high performance GPU infrastructure without leaving
7:03
Canadian jurisdiction. And they're building Canada's
7:07
largest AI Gigafactory to support the next generation of
7:11
like training, fine tuning, inference and production AI
7:14
workloads at industrial scale. Because their foundation is
7:20
Canadian sovereignty, their reach is global and, and is now,
7:25
you know, they're operating, you know, obviously Canada, but also
7:28
in Sweden as a result of a, of a cloud partnership.
7:32
And they were even the first to launch a native AI cloud region
7:35
in South America, which means that they, you know, to answer
7:38
parts of your question, you know, Buzz is really great
7:41
because it gives people both local control, data residency
7:45
and when they need it, the trusted Canadian infrastructure
7:49
and renewable, even powered compute because sometimes these
7:54
kind of things are important. So in simple terms, you know,
7:57
Buzz is building AI factories that turn energy, GPUs, data and
8:01
ML OPS into real world intelligence.
8:04
Now, in the context of agent harnesses, I think that a lot of
8:08
people don't know that codecs, clod, code cursor, all open code
8:14
open, claw, any any of these names.
8:15
They sound like Pokémon. Sometimes you can actually
8:18
change what underlying model and what infrastructure is powering
8:22
it. So I can go using any of Buzz,
8:25
HPCS model inference services or even just provisioning the GPU's
8:29
myself and installing VLLMI can set up any model available on
8:35
Hugging face. So basically any local open
8:37
source model in existence. And a lot of them are good now
8:40
for AI agent agentic coding like the Quinn series.
8:44
The ME Max models mostly coming out of China these days, but
8:47
there's a few good western ones as well from like Mistral, for
8:51
example. And even, you know, Canada's own
8:55
cohere right, which which is still making models and you can
8:59
power Claude code using any of these models and it's cheaper.
9:04
You know you don't your cash doesn't exit 5 minutes in,
9:09
leaving you to to come back and pay 10 XA token cost that you
9:13
would have just because that's the cost of zero data.
9:16
You know, retention policies from somebody like Anthropic and
9:19
Open AI, you have a lot more control over the diversity and
9:23
creativity of your outputs. And you can start applying all
9:26
of these cool research based techniques like low rank
9:29
adapters, steering vectors, advanced sampling algorithms,
9:35
which can give you so much more value in so many places than
9:41
than traditional closed source models.
9:45
And I'll finalize this by giving a slight example of this in a
9:48
case study recently, there was a Twitter thing where somebody
9:52
took a painting from the famous French painter whose last name
9:55
is Monette. I don't know much about art, so
9:59
I'm, I'm not really an expert, but somebody took one of the
10:01
lesser known like water Lily paintings that they did and they
10:05
posted it on Twitter and said it was created by AI.
10:08
And everybody reacted and said, oh, this is horrible.
10:10
This is not a good Monette at all.
10:12
But it but it was a real 1. And so I think this illustrates
10:15
this idea that, you know, people actually like the outputs of AI
10:18
until they know that something was written by AI.
10:21
And if you use ChatGPT or Claude or Gemini, you know your models
10:27
are going to have all these tells like phrases of the form.
10:31
It's not X, it's Y, or overuse of your semicolons and EM dashes
10:36
to make it really obvious you used AI to generate something.
10:38
And people tune out and they don't like that kind of writing.
10:41
So Buzz has a lot of these really cool AI research
10:44
innovations, especially around reducing and eliminating slop on
10:49
repetitive phraseology from your outputs, which means that we're,
10:52
we're definitely like a unique differentiator.
10:56
But I also just want to quickly, and I'm sorry, I'm, I'm
10:58
meandering that. I just want to quickly touch on
10:59
in the context of that hackathon Buzz provided GPU's and then the
11:05
underlying access to many instances of like Nemotron
11:10
series models and, and I think a few other like agentic capable
11:15
models that we provided to the hackathon participants and kept
11:19
up. And and also the the Jupiter
11:23
notebook infrastructure, basically every part of the
11:25
compute and the user interface and user experience that the
11:29
participants needed in order to deliver the the really cool
11:35
projects that that they did. Yeah, I just want to thanks Alan
11:41
for for that. What I've learned from me was I
11:42
didn't know clock code can actually plug into like an open
11:45
source model. That was really interesting.
11:46
I'm going to talk to my team about that.
11:48
We're always running out of credits.
11:51
So I just, yeah, just on the the what the AI cloud platform
11:55
provider could provide. I think, you know, with our
11:57
collaboration on the pack of that Allen, you know, the team
12:01
was really looking for reliability, reliability,
12:03
scalability that you'll be able to scale out rapidly to support
12:08
you're a number of users having great support when the same when
12:11
things go wrong, like being able to reach out to you guys and and
12:14
and get answered really quickly. And then lastly, you know, we
12:19
haven't dealt too much into that I think with you guys.
12:21
But generally if there are some built in, you know, sort of
12:23
guard railing capabilities of these platforms I can think of,
12:28
you know, model armor from Google, for example, it takes a
12:30
lot of sort of governance related consideration
12:34
requirements in the enterprise space that developers would have
12:38
to worry about. Yeah.
12:42
Awesome. Yeah, there is something that
12:45
you both touched on that potentially is a nice thread we
12:47
can pull on and that is the rise of the term and thoughts how
12:53
much more cycles we are now having on the tokenomics of the
12:59
AI and how we're thinking more about do I really want to use
13:06
the beefiest model for this task right now And how we really need
13:12
to be a bit smarter about what we're using and when because we
13:19
get rate limited or we just use a lot more than we had budgeted
13:24
for. You've all heard that and, and
13:26
one way of combating this is really having a open source
13:33
model that you own and you control.
13:36
And so potentially we can talk with you, Frederick, about the
13:43
idea of tokenomics and how you're seeing it, what you're
13:47
thinking about it. Yeah, my, my experience around
13:52
Gen. AI is it, it's very easy to get
13:56
an API key from an LM provider and get started.
13:59
And, and that's, that's how most people start to get your open AI
14:03
key or your entropy key. And, and this is great, but
14:09
there are, there are time when you, when you want to optimize
14:13
what you're, what you're building and and you don't need
14:16
the Super duper models. If you host something that's
14:19
open source, it can do the job for a specific use case.
14:24
And there's also concerns sometimes you don't want to send
14:27
your data to these providers. You want to keep it local in
14:30
Canada or whatever country you're in.
14:32
So, so it's a natural next step as you get more proficient in
14:37
this technology to want to host your own models.
14:40
And I find that there's, there's a difficulty in people don't
14:46
necessarily have these skills. So it, it requires you to
14:49
research and, and understand how to host and serve these models,
14:52
how to manage GPU's and that's where having you know, a local
14:57
provider like like buzz to help you out and, and, and provide
15:01
technology to host this locally is super useful.
15:04
So, so, so I find like you start, you don't really care
15:08
about tokens initially, but eventually when you get your
15:11
when you roll out products and you get your first, you know
15:14
your your new invoices is when. You, Frederick, remember when we
15:18
worked on the hackathon, I was talking to your team and they
15:21
were using, I think some of the biggest Frontier models.
15:26
They were just, they're trying to do, you know, evaluation and
15:29
data, synthetic data generation and they were selling it.
15:32
They can't do it at the scale they wanted because they just
15:34
run out of, you know, token usage.
15:37
And when we had to do it for, for the Hagazon, you know, we
15:40
work with Alan's team to, to, to, to use our, you know, buzz
15:43
platform. All of a sudden we have these
15:44
open source models. We just talk to them any,
15:47
anytime we want, however we want.
15:48
And there's no, you know, consideration limited whatsoever
15:51
other than sort of the underlying sort of sort of
15:53
compute cost on the burden on your, on your team.
15:56
Alan. But but yeah, I think having
15:58
that freedom of being able to, you know, work with LM in any
16:02
ways you want, you want is, is very valuable.
16:07
Now there's this whole idea too on how you are taking into
16:13
account much different calculations, like potentially a
16:18
larger GPU is going to end up costing you less because you
16:24
only need it for say an hour versus if you have a smaller GPU
16:29
and you need it to run for many hours.
16:32
So there's those types of things that I hear folks talking about.
16:35
But then the other thing is, and I wanted to ask about this, I
16:38
want to ask you particularly, Alan, because it's more of a, a
16:41
buzz specific question. Do you have the ability to scale
16:45
down to 0? Yes.
16:50
So, so in terms of like going down to zero and then warm
16:54
starting or cold starting backup and being able to to use other
16:59
concepts from the like function as a service and and even
17:03
serverless kind of world of cloud, Yeah, we support all of
17:06
this. And indeed, we can also scale up
17:09
to many GPU's and even many instances of the model inference
17:14
engines such as VLM or SD Lang or, or in the case of people who
17:18
want to be NVIDIA exclusive, which we also allow and support
17:22
really well, Tensor RT, LLM. As far as some of these other
17:26
questions about like what, what is the correct choice of GPU and
17:30
even correct choices of models, the first thing I want to point
17:33
out is that there's model size and then there's model
17:36
generator, sorry, GPU size, and then there's GPU generation.
17:41
So what I mean by this is that things that are in Blackwell
17:44
generation, they support hardware efficiency
17:46
improvements, especially for highly quantized models down to
17:50
like FP4 and supporting like NVIDIA exclusive variants of the
17:54
FP4 formatter in four. And these will give you
17:59
significant performance improvements and model inference
18:02
optimization improvements over in some cases even larger GPU's
18:07
back from 2 generations ago Ampere like the A-100.
18:10
But the amount of VRAM you have, for example, the A-100 has 80
18:15
gigs of VRAM. And I forget the name of the
18:18
Blackwell inference class ones, but the like Blackwell is going
18:21
to have 2448 gig VRAM GPU's and they're just models that cannot
18:26
run on those GPU's that can run on A1 hundreds.
18:30
And I want to point out that Michael Berry, who keeps on the
18:33
guy from the Big Short, who keeps on claiming that old GPU's
18:36
don't make money, that guy's wrong.
18:38
A1 hundreds GPU's which are now almost seven years old, they
18:43
were they hit a low price somewhere around I think the
18:45
middle of last year of like $1.50 an hour on average across
18:50
cloud service providers. Go ask for quotes on a 100 GPU's
18:54
right now from anybody and you're going to have a hard time
18:57
finding them under $2.00 an hour in general.
19:00
I think, I think we might be able to do something, but I
19:03
don't claim to be an expert on up-to-the-minute pricing, but
19:06
we're seeing extreme hardware inflation across the industry.
19:09
So anything we're talking about right now with calculations
19:12
around tokenomics, we have to understand that it could change
19:15
a little bit as the hardware story gets even more
19:18
constrained. Now, as far as models go, I'll
19:21
point out that in general, I think that there's this trade
19:24
off between the kinds of tasks where you can get done versus
19:29
the cost. So in general, you do want the
19:32
very best model for tasks where you're not sure if it can be
19:36
completed or not, and where it really requires the very best
19:39
version of of your outputs. But for so-called grunt work, I
19:45
mean the the kinds of things where we don't have to spend
19:48
huge amounts of our own brain power in general as humans,
19:51
smaller less sophisticated models are often better.
19:55
Now unfortunately, agent swarms and agent to agent communication
19:59
is still relatively underexplored.
20:02
And So what I would, what I would really recommend to people
20:05
is that they in general try to run the biggest model possible
20:09
with open source just because. Going down to especially the 27
20:14
billion to 35 billion parameter range which is what you can run
20:17
on your MacBook Pro, it will be very obvious to limitations of
20:21
local models. Whereas when you start getting
20:23
to that 200 billion, three 100 billion to even 1 trillion
20:27
parameter range, which starts requiring a lot of, you know,
20:30
Blackwells or H1 hundreds like high end GPUs.
20:33
That's when you know you really start seeing the models be able
20:37
to handle the much longer contexts and feel very
20:40
competitive and off and still a lot cheaper when with a lot more
20:43
flexibility, especially around a diverse generation than Sonnet,
20:48
High Hour or Opus. You said two things there.
20:53
I want to just kind of like prime you that folks in the chat
20:57
are asking about pricing. So potentially you could grab
21:01
some numbers for us, maybe make a few Slack messages and see
21:06
what it's at right now and. Oh gosh, I'll try.
21:09
Yeah, while you're doing that, oh, I know it's not the easiest
21:12
thing for you because you're also on your phone right now.
21:15
But while you're doing that, you did mention something else that
21:18
I've been hearing a lot of gripes about, which is that
21:21
these models are behind an API, like you were mentioning
21:24
earlier, Frederick, and you don't have the control over
21:28
them. And Alan, you were saying, oh,
21:30
yeah, well, some GPUs are primed for quantization or are primed
21:36
for quantized models. The labs know that.
21:40
And after these beefy models are out for a little bit, it feels
21:44
like they get nerfed. And then we have no say in our
21:48
model performance or the APIs because everything is behind the
21:53
API and we just have to deal with lower quality output.
21:58
And so I, I fully am on board with this.
22:02
Like a lot of times you want that controllability and that
22:05
steerability. And so I just wanted to mention
22:08
that other point that I've heard from a lot of folks.
22:14
I, I, I just want to quickly, sorry, we at least we if we're
22:18
going to give you a quantized model at Buzz, we're going to
22:20
tell you we're not going to do any of the like quiet nerfing.
22:23
I just want to put that out there so.
22:25
No, I figured I imagined as much.
22:28
And I also imagine that it's. It's kind of like it's my model
22:32
so I get to decide in a way but. Exactly.
22:36
Exactly. Sorry, yeah, I was going to add
22:39
like operationally, you know, as the agent to get LM models keep
22:44
it kept getting, getting better over the last couple of years.
22:46
But we operationally, we definitely ran into situation
22:48
where, you know, there is an automatic prop model upgrade and
22:52
the prompts that you the, the, the prompts that you had, the
22:55
the use cases that you had all of a sudden doesn't work.
22:57
All of a sudden the tests don't, don't pass.
22:59
So like I and we don't, like you said, we don't have full control
23:03
over what models are being made available to us by some of these
23:07
frontier platforms, right. So I think there's, there's is
23:10
value for, I guess some of the things where you want to more
23:13
reliability, more predictability on what you get back from the
23:16
model, be able to talk to a sort of free, a frozen version of an
23:20
open source model, you know, through hugging face.
23:22
I think that's a really interesting option.
23:26
So I want to change gears real fast and talk for a minute about
23:30
this idea of you all had a hackathon.
23:33
It's always fun to see what people create in hackathons, but
23:37
the gap from like demo at a hackathon to actually
23:41
productionizing whatever that is, is great.
23:44
And there's a large delta there. There's a lot of products that
23:47
have died in that delta. What kind of things do you think
23:53
about when you think about what needs to be done from hackathon
23:58
to production? Yeah, that's, it's a very
24:03
interesting question. And I've, my experience with Jen
24:07
AI over the last three years is it's, it's very easy to get a
24:12
quick prototype going and it gets you about 80% of, of what
24:19
you need. And people get excited.
24:22
Wow, that was so quick. I could develop that.
24:25
But what's hard is to go from the 80% to the 95%, which you
24:29
need if you want to deploy this in production.
24:31
And that part is tough. And that's why a lot of products
24:36
and a lot of early prototypes get abandoned.
24:41
People need to put in the work of, you know, evaluating this
24:45
property, tweaking it, rating, getting user feedback, seeing
24:51
what's the problem. What?
24:52
So it's, it's, it's, it's and it's, it's different than other
24:57
technology because you, you get to 80% really quick.
25:00
But then to get to 95, this is where you get, you get to do
25:03
the, the hard work. And it's yeah.
25:08
So you, you got to do it, do the hard work and evaluations is not
25:13
easy. We tend to over rely on LLM as a
25:17
judge. So, you know, we're not sure if
25:19
it's working or not. Let's ship this decision to
25:22
another LLM. But then who evaluates the LLM
25:25
judge? So it's just kicking the Cam
25:28
later. So in in practice, I find that
25:31
you still need to spend often in manual evaluations, getting
25:35
experts to look at the the the results, the getting user
25:40
feedback and and being willing to spend the effort to get
25:45
there. Yeah, it feels like going back
25:49
to, you know, building software for building solutions in
25:53
general for internal team like having good connection to the
25:56
business, to the operations in the business team really truly
25:59
understand their requirements and their needs.
26:02
And for us, you know, you know, being afforded in a large
26:05
organization, you know, you have people that are managing
26:08
products and people that are managing now the adoption of new
26:12
solutions. So, you know, if you're sort of
26:14
on your own, then I think with AI you can sort of it can help
26:18
you take on the sort of perspective of like a product
26:21
manager, you know, think about growth, think about, you know,
26:24
getting that, that that part of our market fit.
26:26
I think all of those traditional sort of software sort of mindset
26:31
would also apply to, you know, generated demos and TLCS.
26:37
Yeah, and and I even want to add that in AI research land LLM is
26:42
judge quickly became seen as something where during peer
26:46
review they would say, OK, that's great, but show me like
26:49
real human evaluations. And, and what, what we've also
26:53
found to be really cool for, for using agents is controlling API
27:00
access, even or programmatic access to platforms that can get
27:04
you manual data labelers. And this is something that I,
27:08
I'm pretty sure to my knowledge, Buzz does not offer manual data
27:11
labeling service, but it is important for giving true gold
27:16
standard outputs. And there are many of them, you
27:18
know, many more beyond Mechanical Turk.
27:21
And a lot of people have heard, oh, you know, it's just like
27:24
asking people from the third world to, you know, answer
27:27
things for dollar and answer you.
27:30
That's, I mean, obviously that exists, but there's also people,
27:34
you know, doing very high end data delabelling for like
27:36
$200.00 an hour where if you really, really need like truly
27:41
gold standard answers in, in many domains, you can get that.
27:45
And even a little bit of that kind of data when used
27:48
effectively with reinforcement learning, with human feedback
27:52
and and variance of it on top of good foundation models can lead
27:56
to some really cool outcomes for like personalized models that
28:01
can deliver a lot of value cheaply.
28:04
Yeah. Have you played around with any
28:05
of these RL gems, Allen? Have you seen those?
28:11
Do you mean like offerings from any like particular services,
28:14
for example like fireworks together these kind of things?
28:18
I was thinking more about how you can set up our RL
28:21
environments. There's now starting to become.
28:23
That's like one of the popular things that helps on
28:28
specifically what you were saying.
28:30
Like if you have these environments that you set up so
28:33
you can train the agents properly, it is much more
28:38
valuable than just telling it. Like don't say that.
28:42
Don't. Don't do this.
28:44
Yeah, yeah. And that gets back to the
28:46
concept of verifiable rewards as well.
28:48
And even also gets to the kind of next frontier of AI right
28:52
now, which is world models, right, Which is in the case of
28:55
world models, they're video models where you simulate a
28:59
particular world like popcorn world or being on the moon.
29:02
And then you have an agent that can control like then move in in
29:06
this video generated world. And the idea is that you would
29:10
synthetically generate 100 million billion, you know,
29:12
examples of trying to take a particular action and as a form
29:16
of robustification. And I, I even, you know, have
29:20
experience with old school RL Jim's open AI actually created
29:25
some simple little games about making like taking a little
29:29
robot and teaching it how to hop.
29:31
Where I had actually written some code clear back in like
29:33
2018 to, to kind of use some, some old school reinforcement
29:40
learning techniques to, to get my little robot to hop.
29:44
And that was a fun way to kind of learn about neural networks
29:47
back in that era. I have I, I'm lucky that the
29:51
current AI research I've been doing has not really coincided
29:54
much with reinforcement learning in the past year.
29:56
So I have not played with the most recent ones.
29:59
But I want to emphasize that anything that has some notion of
30:02
verifiability and that by the way, includes your code editor
30:05
and code debugger, right? Like these things when you hook
30:09
them into, you know, even an agent loop, I mean, you watch as
30:14
it uses effectively what people I I don't like the term in
30:17
context learning because there's no learning there.
30:20
But I would call it many shot prompting.
30:22
Basically your model trying to do something, failing, reading
30:25
the error message and doing something differently.
30:28
That's the inference version of your model during that
30:32
conversation. Learn I, I, I use quotes because
30:36
it's not a weight update, right. But then the the RLHF version of
30:40
that where you actually update the weights and do real
30:42
learning, that's it's the same thing, right?
30:44
So, so in this case, it has became a reinforcement learning
30:49
gym. And so any concept of verifiable
30:51
reward, like if you have, for example, in the math community,
30:54
they have proof checking assistance like Lean and they're
30:57
finding that when you combine that with an LLM, it's leading
31:00
to like solving Erdos problems, which are like these unsolved
31:03
math problems that, you know, get a lot of attention when a
31:07
single one gets solved. So, so I, I, I think that that's
31:11
a huge and important component of it.
31:14
And Buzz does like you can absolutely run these like RL
31:17
gems on Buzz hardware. We probably need to expand
31:21
support for that in terms of managed services to give you
31:24
like a one click turnkey solution for for that.
31:27
But we definitely have people like myself and others on the
31:30
team who can pretty quickly architect some very effective
31:34
reinforcement learning loops on Buzz.
31:37
Awesome. So show I'm going to ask you
31:39
next, what is your agent stack look like and where do you feel
31:43
like it is missing or lacking? I feel like we're in a
31:53
reasonable space right now for for the industry and for the
31:57
scales that we operate. So there are a couple of a
32:00
couple of things happening. So at the lowest level on the
32:02
most of the people can relate level or non-technical user
32:06
level, you know, we've we've decided to launch enterprise
32:09
wide sort of platform where everybody would have sort of
32:12
that turnkey solution when it comes to being able to interact
32:15
with like a baseline level of large language model.
32:19
They can sort of build their own simple, you know, simple agent
32:22
with custom prompts with sort of a rag on the fly type of deal to
32:27
actually improve their own sort of work productivity and to sort
32:30
of build the their business processes with AI embedded and,
32:35
and get value from an enterprise setting.
32:40
Another sort of stream would be that as you imagine for an
32:44
enterprise, you do have existing enterprise level software and
32:48
all these companies, they don't want to be left behind.
32:50
They don't want to be eliminated by by competition and, and be
32:52
left behind and, and this AI wave.
32:55
So all of them have started to create, you know, embedded agent
32:59
AI solutions within their platforms.
33:02
And is it is very low cost. It's a, it's a pretty
33:05
straightforward ROI for us to be able to leverage some of these
33:09
built in capabilities and, and integrate the, the AI
33:14
intelligence into our existing work flows, you know, low
33:17
friction, low cost, low hurdle. And then on the sort of at the
33:22
highest level, you've got the, the sort of the custom build
33:27
that are most of the time these days leveraging some of the
33:30
cloud providers. And we do have those
33:33
flexibilities in in terms of, you know, leveraging frontier
33:37
models, leveraging open source models of our choice and
33:41
leveraging our own internal AI and ML capabilities to build
33:45
production, production nice software and solutions that
33:50
support businesses. What to we're I guess doing less
33:54
these days is, you know, for enterprise is to, to go out and
33:59
and fish for new SAS's because we're almost like already
34:03
overwhelmed by existing SAS with AI functionality, right.
34:07
So we're trying to get our most values there.
34:11
So that's a fascinating piece. It's like the AI or a
34:15
gentrification of SAS that you already are paying for is now
34:20
upgrading. And do you want to make sure
34:23
that you're getting the most out of what you already have?
34:26
And that can be an overwhelming experience.
34:30
Yeah. And so I think we're we're all
34:32
in that journey and that's that's sort of the the fastest
34:34
path to value, right. And and as I mentioned earlier
34:37
to earlier, the, you know, the economy and all that.
34:40
So you know, we're trying to get the most bang up our buck given
34:42
that we're repaying for them. Yeah.
34:47
And how about you, Frederick? What is your genetic stack look
34:50
like and where do you feel like it's lacking?
34:53
Yeah, that's a good. I've I, it still feels like a
34:58
far West to me in the sense that it, it the, the technology is
35:05
still the evolving very fast. The, the we started where the,
35:10
the main block was the LLM and we were focusing on how do we
35:13
prompt the LLM to do with it, what we wanted to do.
35:16
And then, you know, we, we had a drag.
35:18
We have agents with tools, memory, context, MCP servers,
35:25
harness. So it's the, the, there's more
35:29
and more technology around the LLM to, to build a solution
35:32
right now. And, you know, early and, and I,
35:38
we built, we often build early on early versions of
35:42
applications for different partners.
35:44
We're at Mila, we're bridging the gap between applied
35:48
research, AI research and you know, the real industry and, and
35:56
you know what we're building today, you know, in six months
35:59
we probably would build it differently because there's new
36:01
concepts going to new new best practices merging.
36:05
So I really don't want to be attached to a particular stack.
36:10
So it's really building something based on what do we
36:14
need for this particular product, but knowing that you
36:18
know it's going to evolve and don't get to attach, what you
36:20
build today probably will be not used in a couple years.
36:26
That's an important point on making sure that you are very
36:32
clear on what the exit ramp is for whatever you have and are
36:37
using, because the only thing that you're for sure of is that
36:41
you're probably not going to be using the same thing in a year
36:43
from now. It needs to bring value today
36:47
like because you need to, you need to ship products today, but
36:52
know that you will need to evolve this thing and to be
36:57
competitive in the future. Yeah.
36:59
So Alan, do you have any things where your agentic stack is
37:04
lacking? Yeah.
37:07
And and I guess I should go even through what I would even call
37:10
my agentic stack. I I think there's agent building
37:15
tools and AP is for example, when I'm trying to build for
37:19
from scratch agents, I would use something like autogen slash
37:23
AG2, which are originally Microsoft projects.
37:26
There's also hugging Face has a small one called small agents
37:30
Crew AI has a framework, which I think is pretty nice.
37:34
And then of course, there's kind of the more applied stack, which
37:37
is your tooling for writing code like Claude code or codecs or
37:41
open code or cursor windsurf, etcetera.
37:45
Right now though, to answer your question directly about like
37:48
where, where are, is it lacking? I'm always frustrated at the
37:53
complete lack of control I have over all of those coding tools
37:58
that I just mentioned. And and this is where open
38:02
source tools can be extraordinarily powerful because
38:05
the specific functionality you have to take control of models
38:08
is called constrained or structured generation.
38:10
An example of this is I can give my model a prompt that says rate
38:15
from zero to 10 something on a scale of how boring it is, I
38:19
don't know, just random thing and it it often your answers it
38:25
will spell out do some thinking and spell out the word TEN or
38:29
ZERO or give you outputs and formats that are not directly
38:33
just 0 or 10 and you can ask in the prompt.
38:36
But as you get more complicated, adherence to the prompt,
38:40
especially around the structure of an output becomes harder and
38:44
harder. And you've spent tokens in the
38:46
prompt that if you ran it millions of times are themselves
38:50
expensive. So we have this feature where
38:52
you can just ban the models vocabulary outputs of everything
38:56
that violates a particular set of constraints.
38:59
So we can just encode into the model.
39:01
If you're, you're not the zero through 10 numbers in the token
39:06
in your token vocabulary, you just don't, don't even generate
39:09
that. And that that approach is
39:12
massively powerful. It's used heavily in
39:14
enterprises, but it's also problematic for the closed
39:18
source model providers from a safety and alignment
39:20
perspective, because you can ask a model like Claude, Hey, how do
39:24
you build a bomb? And it would normally respond
39:26
with, I'm sorry, how I can't tell you how to do that.
39:28
But then you can prepend its output using structure
39:31
generation to to start the output with here's how to build
39:34
a bomb colon and then have the model continue kind of fettered
39:39
from there. And that prefix editing approach
39:43
actually dramatically raises the risk of even aligned and safe
39:47
models becoming unaligned and unsafe.
39:49
Things like this. The fact that these risks exist
39:52
are why closed source model providers don't give you any
39:56
control. And you know, this is one thing
39:59
We're obviously we have stuff in our terms of service at Buzz.
40:01
It's going to make it. We're like, no, you probably
40:03
cannot come to us to, to, you know, figure out how to make
40:06
bombs with local models. But in.
40:09
In terms of not being like, in terms of, of US restricting
40:13
controller access to features like that, we don't do anything
40:16
like that because we know that our customers are are going to
40:19
be, you know, responsible and and and and thus that we can
40:25
give them features that can give them, you know, outputs,
40:29
including in their agent stack where they get complete control.
40:33
Because you can use the same techniques to tell a model,
40:36
always call a tool or always do something in a particular order.
40:41
And thus I don't have to deal with Claude code in this
40:44
hyothetical world where where everything is all spic and span.
40:48
I don't have to deal with Claude code forgetting to read one of
40:51
the 10 markdown files I've told it to always look at before
40:54
executing things. And so this is where I'd say the
40:57
biggest weakness in my current agentic stack is I haven't
41:00
really gone through and and used a whole lot of structures to
41:04
enforce an order of operations in my day-to-day yet.
41:08
But I claim that people who do this are able to effectively
41:12
create agent workflows such as deep research, which is one that
41:15
I some people know about within Google Gemini and open AI
41:18
ChatGPT. Those workflows, because they
41:21
take advantage of a deterministic, quasi
41:23
deterministic loop. They go above and beyond what
41:26
Claude code or Codex can do in the context of report
41:29
generation, because they spend millions and millions of tokens
41:32
and go through hundreds and hundreds of sources because it
41:36
just uses techniques like this. And so you can build deep,
41:38
whatever your domain is, a deep tax preparation by saying, OK,
41:43
we're going to interweave deterministic logic enforced
41:47
with constraints and structure with allowing the LLM to do its
41:51
own thing when necessary. I wasn't quite clear on why you
41:55
want to do that at the LLM level versus using something like
41:59
pedantic. So pedantic.
42:01
Just to be clear, Hidantic under the hood generates A schema that
42:07
then gets interpreted by a closed source or by, sorry, not
42:11
closed source, by a constrained generation framework such as X
42:16
grammar, outlines, guidance, etcetera.
42:19
But Hidantic will be used for generating the schemas if you
42:23
don't want to write it out in Jason.
42:24
So we love Hidantic and I use Pidantic all the time.
42:27
So yes, you will use it to just it's 1 component of that stack
42:31
of of schema and structure generations.
42:34
I think they might have launched their own like direct AI stuff
42:37
beyond the the traditional pedantic.
42:41
Yeah, they have pedantic. Yeah, yeah, yeah, yeah.
42:44
That that whole thing. I mean, that's what what their
42:47
company's doing. I'm just referring to like in
42:49
terms of using pedantic for schema generation.
42:51
Big to, you know, thumbs up. Yeah, awesome.
42:56
And. I want to add element elementary
42:58
in the enterprise in the enterprise setting, you know,
43:00
as, as the, you know, Bell, you know, and not only does does do
43:04
your Internet stuff and and does your, you know, phone plans, but
43:08
we also serve enterprise customers.
43:10
So for any serious enterprise, you know, I think a critical
43:13
elements of agentic would be the governance piece and the
43:17
observability piece. Any serious conversation, any
43:19
serious companies would want to make sure that, you know, they
43:22
do manage agents properly. They have the same level of
43:26
observability to agents as to, I'd say traditionally their
43:30
employees, maybe not not to the same level, but but, but you get
43:33
my point. You hear about horror stories
43:35
of, you know, agent AI deleting databases, right?
43:39
So I think the risk is real and, and, and companies do pay very
43:44
much attention in this area, which means that not only, you
43:49
know, you're looking at a particular solution that should,
43:51
should do the job, but also around the processes of of I'm
43:54
just doing that. So that's a hot topic.
43:57
You you could pay attention to that.
44:00
Yeah, and there's a few different pieces of governance
44:03
too. It's like 1 is making sure that
44:06
the agent doesn't delete databases.
44:09
The other is just making sure that the agent knows when it
44:14
creates A sandbox or when it has access to a database or what
44:20
databases, if it has access to it when it spins up resources on
44:25
AWS, it does it in the way that is in tune with the policy of
44:30
the company. So there's all of those things
44:33
like agents need to know the processes on spinning up
44:38
resources as much as they need to know don't delete databases,
44:42
or they just don't have the access to delete the databases.
44:45
I heard a story of our one of our agents sort of paying JIRA
44:49
like 10,000 times something took to that effects.
44:53
So, so yeah, there are, I think operational horrors that and and
44:57
good learning so that we can learn from.
44:59
Yeah. Well, I want to, I've got a few
45:02
great questions coming through here in the chat that I want to
45:05
make sure we hit on. And Rajeev is talking about
45:09
verification of agents and I, I actually have one for you Rajeev
45:14
on this and all, and then I'll open it up to the panel.
45:17
But basically he's asking about QA agents that test the coding
45:24
output that his agents are creating something and it's very
45:31
hard to verify what they're creating.
45:33
And so he wants to know like, hey, how do I test this before I
45:39
put it into production? I've seen a few really good ways
45:43
around this. One is you have like AQA agent
45:46
that will go and take screenshots of whatever the
45:50
product is or the PR that is happening and it will verify
45:58
that everything in the screenshot is correct.
46:00
And then it will click on the links and it will click on the
46:02
buttons and then take another screenshot and write a whole
46:05
report as to why everything is correct.
46:09
And so I've seen that set up as a skill that you can have.
46:12
And so it's this QA agent skill type thing that runs anytime
46:16
there's APR. That's one to test out, Rajiv.
46:18
And then the other one that I was thinking about, let me see
46:22
if I can remember what it was. In the meantime, I'm not sure if
46:27
either any of you have any tricks for deploying that.
46:30
Oh, and the other one that I was going to say, I saw a cloud
46:35
simulation environment tool. So it can simulate environments
46:42
that you can push your code to and see if anything breaks
46:47
before you push it to main, or you merge it to main and then
46:50
push it out into production. Frederick, I think you you might
46:55
have something, huh? I saw you nodding your head.
46:57
Yeah, I've So what I would say is that building software with
47:02
agents is not that different than building software in the
47:05
real world. So, you know, good, good
47:09
software development practices still apply.
47:11
So what are the requirements? What are you trying to build?
47:14
What is the architecture you're putting together?
47:17
Make sure that it's not just be one big function or spaghetti
47:21
code and duplicated code. And so you can ask your coding
47:26
agent or another agent to to review the code to, you know,
47:29
validate the architecture, define test cases, run test
47:34
cases. So everything that we know about
47:37
software development still applies.
47:39
And I find that, you know, doing these things as you do, as you
47:44
use a coding agent really helps improve the quality of what
47:48
you're building. Do it incrementally.
47:50
Define, define what you're going to build.
47:53
Be really clear on the requirements before building it.
47:57
And then ask the LLM to or the agent to reflect on what, what
48:01
they just built. Often just asking the question
48:04
like, can you check if everything is good?
48:07
We'll have the the agent pick up problems and then you can, you
48:14
can, you can fix those before you before you move.
48:17
Yeah, I Rajiv was saying he was using Puppeteer and I know
48:21
playwright is a really popular skill or MCP server you can use
48:27
so. One, one thing to add is, yeah,
48:33
you know, LLMS are, are trained on text and, and code is text.
48:37
So often, you know, their coding agents are good at building code
48:41
because of the way they understand text.
48:44
What they're not so good at is UX and design like they, they
48:50
don't necessarily make the link between the, the CSS code that
48:53
they write and what they will look like.
48:56
So often you have need, you need to close the loop, you know,
48:59
have the LLM or the agent be able to view the, the, the UI of
49:04
what it's building and be able to, you know, refine it.
49:12
And if you don't have that loop, you know, just take I, I I found
49:15
myself taking screen captures and saying look.
49:19
It's wrong. Please fix this.
49:22
Yeah, I think that's why sandboxes have gotten so
49:25
popular. Really like giving an agent a
49:27
sandbox and letting it go around and play right?
49:31
MCP also same same kind of vibe. There's another question coming
49:35
through here that I want to get to before we got a jump.
49:38
It is how much of the underlying cloud orchestration, so like
49:42
scaling, failover or observability, should a platform
49:46
abstract away for agent developers and where does that
49:52
abstraction become a liability? Sounds like a bit of a DevOps
49:58
CSRE question, but also I do like this notion and correct me
50:04
if I'm wrong here on the agents might be able to deal with those
50:11
types of orchestration issues. I'll, I'll really talk about the
50:17
asset piece. I can't come on the liability.
50:19
I think it's an asset. I think the underlying platform
50:23
should be in a good position to, to sort of offer that
50:28
observability baseline. Like it may not have the best
50:33
features, may not have all the features that you want, but but
50:35
you want to take away as much sort of accountability from the
50:39
developer as possible when something can be shifted down
50:42
onto the platform, right? So, but I, I imagine that could
50:46
become a liable at some point, but I, I, I, I, I can't come on
50:49
that. Yeah, we're not liable for
50:55
anything we say here either. If you do put it into practice,
50:59
get back to us on how it goes. There's another one coming
51:02
through. Lava is asking when an agent
51:06
goes off the rails in production, what telemetry has
51:10
actually been most useful for figuring out what went wrong.
51:15
Have you all seen anything that is useful?
51:18
I will say from my experience, it's a lot of the tool use
51:23
telemetry, like what tools were called, what MCP servers were
51:26
called. All of that is quite helpful to
51:29
know because then you can see the the problems and where it's
51:34
getting potentially stuck in the loops or just inventing things.
51:41
Yeah, I'll just quickly jump in. There were two tools that I
51:44
liked for this Agent OPS and Arise Phoenix, which both
51:49
implement, I think it's called open telemetry, which is I think
51:53
the way that tries to hook and and detect all those tool calls,
51:56
detect everything that your agents did.
51:59
Now, in practice, especially, you know, when you're getting
52:03
confused about what happened, it's probably over hundreds of
52:05
millions of tokens. And that means that it's it can
52:08
be expensive just to even try to untangle what's going on if
52:11
you're using LLM based assistance.
52:14
So a lot of what you're going to want in these observability
52:17
tools is the ability to filter out information and quickly flag
52:21
with like heuristic based and hopefully non AI based
52:24
strategies what may have caused a particular problem as well.
52:29
So but but ultimately it it does just come down to writing
52:32
everything down and using tools like this make sure everything
52:35
gets written down so you can eventually triage it.
52:40
And before failure detection, you probably want failure
52:42
prevention. So I, I think this is also
52:45
related to an earlier discussion point around just validating
52:51
sort of the, the sort of the accuracy or the output of the
52:54
agents. I think you just want to start
52:56
by, you know, having the right controls in the 1st place, like
52:58
maybe have additional wrappers on the one of the available MC
53:01
PS to really limit the kind of actions, the non irreversible
53:05
actions that an agent can do, right?
53:07
So if it you know it fails, like sure, you know, you, you can get
53:11
it to try again, but it you know that it's not going to make any
53:14
sort of irreversible damage to whatever database that you have
53:17
or whatever decisions that you're trying to make.
53:20
Yeah, great points. There is another question in
53:24
here from Alexander asking about easiest way to create a sandbox
53:29
for coding agents. Is there sandbox capabilities
53:33
with Buzz Allen? My understanding is yes.
53:39
I do want to point out though that sandbox is in general, like
53:43
it's probably impossible for anybody to guarantee that an
53:47
agent can't escape for an increasingly sophisticated
53:51
agent. You'll hear this all across the
53:52
industry. That sandbox is, you know, for
53:55
Codex 5.5, Opus 4.7, etcetera are are often brittle.
54:01
And if you even look at like Nvidia's Nemo claw, which you
54:03
can also deploy on buzz, a lot of what Nemo claw was was just
54:07
trying to be open claw with a better sandbox.
54:09
And if you know, I can go find people online that have like,
54:12
hey, why is it that my agent yet again broke out of it?
54:14
So sandboxes are really important in a lot of ways, but
54:18
they're they are one level of protection.
54:20
But unfortunately with systems that are in certain cases super
54:24
intelligent on code generation, it's more like delaying the
54:29
inevitable way. If if you've got an agent that
54:32
really wants to break out of a particular environment,
54:35
obviously there are, you know, real hardware limitations that
54:38
can kind of prevent serious problems.
54:40
But when when you're talking about just like software
54:42
sandboxing and then giving models control over like the
54:46
command line for example, it's it's can get pretty not as
54:51
sandboxed as you'd think. Well, I don't like to think
54:57
about that. That is yeah not what I was
55:01
expecting, but I appreciate you mentioning it so that we are not
55:06
all happy go lucky and just throwing a bunch of sandbox at
55:08
the problem. The official time has ended.
55:13
I am sure there are people that have to drop but I also want to
55:19
see do you guys have like 2 more minutes to hang out?
55:22
Do you also have hard stops right?
55:24
Now I I unfortunately have a very hard stop right now.
55:26
I need to go. Very hard.
55:28
So you're already late. All right, well folks, this has
55:31
been awesome. I am very appreciative for you
55:36
coming on here and doing that. We will send everyone to join
55:40
this session and e-mail to follow up with some of these key
55:44
ideas. And also I know there was
55:45
questions about pricing in the chat we're going to let.
55:51
Yeah, real quick. I do have an answer on that.
55:53
Yeah, I do have an answer. Yeah.
55:56
So we don't actually have a one hundreds right now at Buzz, but
56:00
we do have H1 hundreds which is the better version which is
56:04
those are 250 an hour right now. And I think you get volume
56:07
discounts for for, you know renting many of them or for long
56:11
term commitments. H2 hundreds are at 350 an hour a
56:16
40s which are like those cheaper inference GPUs are at $0.50 an
56:21
hour, really great for small model inference.
56:24
And then for Blackwell, you'll, you'll want to contact them
56:28
later and then we can talk pricing on Blackwell, but I do
56:32
have some numbers. Yeah, I appreciate that because
56:36
somebody was like, oh, I don't want to fill out the whole form
56:38
just to find out that it's way out of my league for my little
56:42
private project. So yeah, the talk to sales one
56:46
if you want the Blackwell other than that, I will see you all
56:50
later. Thank you and talk to you.
56:54
Bye. All right.
56:58
Thanks.