Latest / Agentic Conversations (formally mlops.community) / Exploring the Impact of Agentic Workflows // Raj Rikhy // #268

Transcript

Exploring the Impact of Agentic Workflows // Raj Rikhy // #268 · Agentic Conversations (formally mlops.community)

0:00
My name is Raj Ricky, I'm a principal product Manager at
0:05
Microsoft in Azure Data and I usually do shots of espresso
0:13
over ice with oat milk. Agents, agents, agents, agents,
0:18
agents. Today we are talking agents and
0:22
it's just in time because our next virtual conference is
0:25
coming up on November 13th and guess what that's going to be
0:28
about? Yep, AI agents in production.
0:34
It's not just agents, it's agents in production.
0:37
So if you want to join us, we'll leave a link to sign up as we do
0:43
in the description. For now, talking with Raj, I had
0:48
one gigantic takeaway and it was around the question of how do I
0:53
know if something is not possible or if I'm just not
0:57
using the right tools to accomplish the task?
1:03
Do I need to try a different agent framework?
1:06
Do I need to try something different in my stack to get the
1:11
desired outcome? And his response, I thought was
1:17
right on the money. But's here.
1:19
But before we do, I've got to play you a song just to get that
1:23
recommender system that you're rocking, a little bit of
1:27
variety. People enjoyed the Grateful Dead
1:31
hit that I played the other day. And so in that same vein, I'm
1:36
going to play you one of my favourite songs from a jam band.
1:42
This is fish and this is Farmhouse right out the gate in.
2:06
So some of you may be saying, ah, Fish.
2:09
Really Trey? He's not the coolest guy.
2:15
Some people don't like him. I hear you.
2:18
But you can't deny this is a jam right here.
2:22
Let's hear it. I never.
2:25
Ever To we'll start with this because
3:13
it feels like there's the agents that you can have that are
3:17
inside of certain programmes that will work that programme,
3:22
but then there's like open-ended agents.
3:25
Then there's agents like I just said, that they're dealing with
3:27
pixels on screens. So when you look at the plethora
3:31
of agents that are out there, how do you categorise them and
3:34
bucket them? I think it's helpful to take a
3:37
step back and before we talk about like, because we do want
3:41
to talk about it in the context of actions, but let's let's take
3:44
a step back and figure out like, what are we actually talking
3:46
about here, right? So agents actually, as a matter
3:51
of practise have been in, in the academic discipline since,
3:57
believe it or not, the 50s. Like reinforcement learning has,
4:01
you know, the field of reinforcement learning in both,
4:03
you know, things like psychology as well as things like dynamic
4:07
programming. It's, you know, it's really been
4:10
about understanding within the scope of trying to take actions
4:15
on the user's behalf, right? What is the, what is the
4:18
universe of things I can do and why and why do I want them, you
4:22
know, to do those things and how do I make them do it better,
4:26
right? Like these are actually the
4:27
principles that underlie when we talk about like agents, forget
4:31
about AI when we talk about agents, right?
4:33
These are very fundamentally the things that, you know, we look
4:37
for and, and if we even peel back another layer, like what is
4:41
an agent, right? Forget about like what we're
4:43
trying to do with them. What is an agent?
4:45
And at the at the base level, an agent is some kind of an entity,
4:51
software or otherwise. It's some kind of an entity that
4:54
takes actions autonomously, that has decision making power and
4:59
capabilities. That's what an agent is.
5:03
And I think the key thing here, to your point around
5:08
differentiating agents, if we think about an agent as some
5:12
kind of an entity that can act autonomously, the reason why
5:15
it's useful to understand the lineage, the provenance, the
5:19
history of agents is that we can learn things that have been
5:27
studied since then. So in reinforcement learning,
5:31
when you're using agents, there's just by virtue of the
5:35
fact that like we're trying to get things that we've been
5:37
trying to get things done for a long time.
5:39
And just for some context, like, you know, I was part of the
5:41
product team that led developing the first deep, you know, deep
5:45
reinforcement learning platform as a service for Microsoft.
5:48
Like the first one that really achieved a, a high level of
5:50
scale. And, and, and our customers were
5:53
asking, you know, at that time there, there's not really a
5:58
whole lot of folks that are, you know, I always say it's being
6:04
done in research, but it's, it's not really like, oh, OK, you
6:07
know, I'm going to go deploy in my business and kind of a thing.
6:10
And there are these principles that we kind of had to educate
6:14
folks on. And I think it's worthwhile to
6:16
talk about them, right. So one of them is we talk about
6:19
our environment. We kind of sit pen to paper and
6:24
figure out what are, what is the universe that we're operating
6:27
in? Like we're not even talking
6:29
about, you know, start coding things up.
6:31
We're not even talking about like what model you're using.
6:34
I mean, this is like verboten stuff at this at the moment.
6:37
We just want to get a sense of like, what are we trying to
6:39
accomplish? And when you sit down and, and,
6:43
and begin the process of, of ideating, you know, what is your
6:47
actual end goal and, and what is the intended behaviour that you
6:51
want the agent to undergo? You've taken that very first
6:54
step and really it literally as simple as sitting down.
6:58
And, and This is why I think some PMS are drawn, you know,
7:02
and, and even data scientists are capable of like visualising
7:05
those products they're drawn to try and figure out like, OK,
7:07
what I want to get done with agents because in a very real
7:10
way, what you're doing by practise of writing things down
7:12
with agents is you're drawing, you're creating an MVP for
7:14
agent, OK. And, you know, when you think
7:19
about like, OK, what are the different types of things I can
7:22
do with agents or what are the different types of agents, it
7:25
really is a function of, OK, well, now you've written down
7:28
what it is that you want. Now figure out how you're going
7:31
to get it done. And so whether that means in the
7:36
context of a video game, right, your purpose agent is to be a
7:42
enemy, to be an antagonist and light the light, the main
7:47
character on fire, right? Like that's, you know, that's
7:51
your purpose. That's what you need to do.
7:52
Figure out a way to do that. OK.
7:55
And that's one way to think about it, another way to think
7:59
about it in terms of what you want to accomplish and how it
8:01
is. It's like, OK, how am I going to
8:03
figure out how to get this guy to light him on fire?
8:04
Well, he needs a flamethrower, right?
8:07
Obviously he needs to figure out how to understand the
8:08
environment. He needs a map, right?
8:11
And, and these are tools. This is how an agent
8:13
accomplishes its goal. It's a tool, a tool set, right?
8:16
And, and, you know, practically speaking, when we talk about,
8:18
you know, agentic implementation, we talk about in
8:21
the form of it's either a function call, right, or a, or a
8:24
tool call, right? But ultimately, you know, if
8:27
we're connecting the dots here, that is the capability of an
8:31
agent, right? You're describing the things
8:33
that they can do. They can drive to the grocery
8:34
store, they can pick up the bag of flour, right?
8:36
These are the things they can do.
8:38
So you've gone from defining your goal to defining what it is
8:42
that they're capable of and how you can offer them things to do
8:45
that. OK.
8:47
I just want to pause for a second because I think you said
8:49
something that is worth noting too.
8:52
On the environment piece. If we're talking about the game
8:56
the agent is playing in the game, that is their environment.
8:59
If we're talking about an agent looking at pixels on a screen,
9:02
that is their environment. If we're talking about an agent
9:05
working inside of Microsoft Fabric or Microsoft Office, that
9:09
is their environment. And so almost like you're
9:11
putting the guard rails on, here's your environment, here's
9:14
where you play, and this is one other piece, one other
9:18
principle. Now we'll give you the tools.
9:20
Now we'll give you the why or what we want you to accomplish,
9:24
but it's going to be within this framework.
9:27
Right. And I think that that's
9:28
important not just because, you know, you're thinking about the
9:33
universe that they're playing in, not just because you're
9:36
thinking about like what capabilities they have and what
9:40
what they can get done. There's this other principle
9:42
about agents that, you know, in the scope of what we're talking
9:45
about is really important, which is how well can they do it,
9:50
right? And in reinforcement learning,
9:54
we ascribe this principle basically around reward, right?
9:59
Like folks are kind of familiar with, you know, positive
10:01
reinforcement, negative reinforcement, right?
10:03
Like you want to do positive reinforcement because you want
10:05
to continually reinforce the, the positive things and what's
10:08
the reward? The reward of positive
10:10
reinforcement is praise, right? Like that's how you offer that's
10:13
a that's quite frankly the reward you're, you're offering
10:17
the increase in praise, which will lead to another outcome.
10:19
Well, in you can't necessarily praise software, although you
10:22
can try, right? But but what you can do is you
10:27
can define parameters for success.
10:32
So in a game that is as an example, like let's use our
10:36
flamethrower guy or flamethrower gal, right?
10:39
Light him on fire more, you know, decrease their health
10:42
faster, right? Get them, you know, zero to fire
10:47
in less than 3 seconds. You know what I'm saying?
10:48
It's what are the parameters for success.
10:51
Make sure you deliver that. You know, it's these parameters
10:53
of success that matter in the scope of accomplishing an agent.
10:57
Once you honestly, you do not need to programme anything to
11:04
accomplish what I just said. And that is important because as
11:09
you know, as we know, programming and and and code
11:13
development is a meets to an end.
11:15
Yeah, right. Like it is a tool that is used
11:18
that has, you know, other types of libraries and and you know,
11:22
anything that you want off the shelf to go and accomplish
11:25
something. But you don't necessarily need
11:29
to programme to accomplish the objectives of a successful
11:34
agent, whether it's in the scope of a game or scope of Anyway,
11:37
I've been talking a lot, but let me let me pause there.
11:40
Does that does that make sense? Is that us?
11:42
Yeah, for sure. And one thing that came to mind
11:46
when you were saying this is the trade off, I think that you get
11:52
between defining how narrow you define the success criteria and
12:01
how much you leave open for interpretations.
12:05
And I can see how sometimes we we can inadvertently shoot
12:09
ourselves in the foot by defining too much.
12:12
That's right. But I also see that you want to
12:15
define more because it gives you a little bit more reliability.
12:19
So there's that trade off, there's that line that you have
12:22
to walk. You know this is so important,
12:25
right? Like this is where you know the
12:31
pedal meets the metal in terms of agentic development.
12:34
Do you over define? Do you not over define right?
12:39
And the the tools that we use to make these determinations are
12:44
getting more sophisticated, more mature and more abstract, which
12:48
is a good thing for everybody involved, right?
12:51
I think about six months ago, you know, as an example, the
12:55
most popular or one of the most popular repos, auto GPT.
12:59
They had this interface called Forge.
13:02
It was like this UX and they, they might still have it, but
13:06
the point of it is that, you know, it basically functioned as
13:08
this like high level abstraction where you can put in some text
13:11
and the text becomes the meta prompt.
13:13
And then, you know, the, the agent is supposed to be defined
13:15
by that, right? But it's not, there's no like
13:19
it's, it's not Facebook, right? It's not exactly the easiest UX
13:22
to work with. Right.
13:26
And then, you know, there's this other step level thing of now,
13:29
now we have some nicer interfaces, now we have, you
13:32
know, cleaner libraries and now the code is more mature, there's
13:34
more pull requests and things like that.
13:36
The other thing that's at play here is, and this is kind of a
13:40
big word, but it's used intentionally, like is, is your
13:43
orchestration right? Like what is your scheduler who
13:47
is doing the planning, right? Are you offloading that planning
13:51
to somebody else in the form of like Azure Open AI is doing the
13:56
planning for you because it has a platter behind the scenes and
13:59
it's doing that on your behalf or, you know, open AII say Azure
14:03
Open AI, just because it's a lot of, you know, that's an
14:05
interface like Open AI has an assistant API, right?
14:09
You know, there are other clawed as an example, has a function
14:13
calling API, right? And it has a planner to figure
14:15
out how it calls those functions.
14:17
And so a lot of that logic is there, but if you're running it
14:19
by yourself, you have to figure out what that plan looks like.
14:22
So whether you're defining it like semantic kernel or Lang
14:25
chain or land graph or whatever it is, that's your
14:27
responsibility. And you know, if you're not
14:30
coding it, are you leaving it up to an algorithm, right?
14:32
Like they're planning algorithms out there too that can do this.
14:35
And maybe those are good for you.
14:37
The the point of it is, you know, the definition that you
14:40
put into the environment and the criteria for success should
14:45
start upfront on paper and you should have a well oiled
14:51
understanding of how success looks like before you get to the
14:55
point where you were running wild.
14:56
And by the way, there's one more thing that I want to come like
14:59
just call out here. LLMS are great, right?
15:05
But LLMS are random and that is by nature, right With there's
15:15
about a year ago, you know, and you probably heard this before
15:17
Demetrius, but like, you know, people are like, Oh yeah, we
15:20
ought to reduce hallucinations. It's hallucinating all the time.
15:23
It's always hallucinations are the product.
15:27
That is the product, right? Like that is what an LLM does.
15:31
It fabricates your your goal, right?
15:35
Your goal is to figure out how to effectively fabricate and I I
15:39
know that sounds ridiculous, but by definition, a large language
15:43
model is probabilistic and it's stochastic.
15:45
It is designed to be read OK. It is.
15:48
You can create affordances. You can create capabilities that
15:55
are more deterministic, that are more planned.
15:58
But that is up to you to define and to govern as the quote,
16:04
unquote. And I'm saying this only
16:06
because, like, it's important to think of yourself this way as
16:09
the architect of that system. Yeah, right.
16:12
You may not be an architect. That's OK.
16:14
You don't have to be an architect.
16:15
What you do have to think about is that you are acting as an
16:19
architect. You are, you know, the puppet
16:21
master. You are pulling the strings.
16:23
That is who you are. Whether you want to accept that
16:25
role or not, it is important because anything that happens in
16:30
that system, good or bad, ultimately are things that you
16:37
can affect. Not control, but you can affect,
16:39
right? Because remember, agents have
16:41
autonomy. The idea here with defining all
16:45
these different pieces and recognising how many variables
16:49
you have and that you can, how many knobs you can tweak, right,
16:55
is is very valuable. And so maybe it's, it's worth
16:59
talking through different knobs that you want to be looking at
17:05
and focusing on or ways that you can.
17:08
I think I I'm thinking about it as how can we?
17:14
For my own use, I've gone and pounded my head against the wall
17:20
because I don't know if what I'm trying to get done is not
17:25
possible or if it's just I'm not able to make it work because of
17:31
my architecture or my puppeteering as you will.
17:35
And so I'm wondering if you've seen any good ways to just get
17:39
that fast feedback loop and recognise and debug quickly?
17:44
And, and that's a excellent point because like I said,
17:47
right, if you are in a position where you are architecting that
17:53
system, you want to figure out what failure states exist before
17:59
your users or whatever, you know, forget about the users.
18:02
Like let's say you're not designing for somebody else,
18:04
they're designing for yourself, right?
18:05
You want to figure out what those failure states are before
18:08
you put it out in the wild, right?
18:10
And you're never going to be able to discover every single
18:13
one. But there are some tried and
18:15
true patterns to help you figure that out, right?
18:19
And I'll give you an example of how not to do it before I tell
18:24
you how to do it. Like one of the things, you
18:27
know, people, again, we're talking about the hallucinations
18:30
thing because people tend to be like, you know, people tend to
18:33
be angry about it. They're like, I don't know why
18:35
It's so, you know, making fake crap up.
18:39
I don't get it. You know, it's I just wanted to
18:42
write a legal brief. Stop making up fake law cases
18:46
that you know, it's like, OK, listen, listen, right.
18:50
The the way that you know, the way that you can explore what
18:54
these failure states are is actually pretty straightforward.
19:00
Here's pro tip, never give your agent raw web access
19:07
out-of-the-box. Just don't do it.
19:11
It's a bad idea. It doesn't matter what
19:14
safeguards you put into place. Just just don't do it right.
19:20
You need to give it some kind of directive, OK?
19:24
Like, it's dumb. Yeah.
19:27
Too much agency. Too much agency?
19:30
Yes, yes, it's like it's like here's a tablet 5 year old, you
19:34
know, go get go get addiction, find out.
19:38
Yeah, yeah, probably. Yeah, gosh, you know, I found
19:42
YouTube, you know, and it's like the the key here is vet the
19:49
outcomes yourself 1st. And we always, you know, even
19:54
before agents came along, we would say this like with
19:57
generative AI always at the 1st generation for the MVP, keep the
20:01
human in the loop. Agents are designed to act
20:04
autonomously. That does not mean that they
20:06
have to action that autonomy. You have the opportunity to
20:10
stopgap before that action is taken and ask it to explain the
20:16
next steps that it is in fact going to go do and in fact.
20:19
Actually, it's quite good at outputting the text kind of in
20:22
like a debug log or a debug stream.
20:24
What it's going to do and what it's supposed to do.
20:27
You could ask it to simulate its own activities.
20:29
So that's one way to do it like a human in the loop
20:31
interpretation of of what it's actually supposed to do.
20:34
Another way to do it is to constrain the state space,
20:39
constrain the environment, right?
20:41
See how successful you are with a small landing space before you
20:46
go out and do a broader thing. And I'll give you 2 examples of
20:50
this. Like the first is, you know,
20:53
yeah, I have kids. I'm assuming I don't know you
20:54
have kids, but you mentioned A5 year old.
20:56
So you have kids. OK, So if I give my five year
20:58
old chocolate milk, the first thing that he's going to do if I
21:01
don't, if I'm not looking, is he's going to go, you know, run
21:04
toward the couch and sit somewhere comfy.
21:06
And invariably what's going to happen, it's going to get all
21:09
over the couch. That's what happens.
21:10
OK. And you're like, OK, I've got to
21:13
clean this up. What do I do?
21:16
Well, you're not going to use a random clear on the most visible
21:21
spots on your couch. You're going to spray it in an
21:23
unaffected area first, make sure it doesn't, you know, mess up
21:27
the whole couch. And then if you're OK, then use
21:30
it on the chocolate milk, right? You don't want to make a problem
21:33
worse just because you think it's going to solve it, right?
21:37
Like the cleaner is the agent in this case, right?
21:40
So as an example, let's say that you want to create a support
21:44
agent. You're not going to give it the
21:48
whole compendium of your support documentation and start firing
21:52
it at support tickets. That is a recipe for failure.
21:56
Instead, instead pick like I would say, cherry pick some of
22:03
the lower priority support cases that are more clearly defined
22:09
and provide it with that support documentation context and
22:13
understand the behaviour of how it would address those and
22:17
whether you have a relatively high NPS on ticket closure or
22:22
not, right? Try and create a smaller
22:26
constrained environment to start with.
22:28
And then the third, you know, tool that I would say for
22:32
agents, OK, is don't try and get fancy out-of-the-box too fast.
22:41
You know, I mentioned there are things like planning algorithms
22:44
and you know, you have an opportunity to like try and
22:47
offload some of that stuff like an MVP.
22:50
If your MVP is pen and paper, then the step beyond MVP is not
22:54
like I'm going to code the second line chain, Please don't
22:58
you know? And the reason why I say that is
23:00
because you want to try as much as possible to understand how to
23:06
action success fastest. So getting stuff off the shelf
23:11
to do that, like, for example, using, you know, a function
23:14
calling API or tool API, or even if you're not a code, you know,
23:19
first person like go find a no code tool after you've done your
23:24
pen and paper and see if that solves your problem.
23:27
And if it's not solving your problem, understand why it's not
23:29
solving it, but don't get to the point immediately where you
23:32
start going. I need to, you know, quite
23:35
literally open up the hood and start, you know, changing my
23:39
transmission before you got took Safeway, right, like that did.
23:44
That makes no sense to anybody and it shouldn't make sense to
23:48
you, right? Like we these same principles
23:50
apply to software development, right?
23:52
You know, you wouldn't start writing your own database just
23:55
because your relational database is not sufficient for your use
23:58
case. It's crazy, right?
24:02
It's literally crazy. Don't do it.
24:04
But people, you know, are like whatever, you know, people like
24:06
just do this. That's fine, right?
24:07
But but you know, there are these things.
24:09
These tools exist for a reason. Don't bring the burden on
24:13
yourself and try and iterate from there.
24:16
One thing that I going along those lines of that last point
24:21
you made and I often think about is you mentioned auto GBT
24:27
earlier, right? Totally.
24:29
If you were playing with auto GBT 6 months ago or a year ago
24:33
and you were trying to use agents and you felt like, Dang,
24:37
this just isn't working. These aren't reliable what's
24:40
going on? And and you were trying to
24:42
tailor your environment and do these things.
24:45
At the end of the day, you probably would land on the
24:47
conclusion of it might be the wand, not the wizard here.
24:53
Like I, I wish I could make this work, but I don't think I have
24:57
the proper tools right? And so then maybe you jump to
25:00
another one. And now today there are so many
25:05
different agent frameworks and rules that you could use tools,
25:09
not in the sense of agents using tools, but us as the
25:13
orchestrators being able to use tools or a product, I should
25:18
call it. How do you know when the product
25:21
has hit its maximum? How do you kind of evaluate
25:25
that? Like oh, this may be because
25:28
this. Productation of this thing
25:31
exactly. I think so that's it's, it's,
25:35
it's useful to evaluate it in the context of the scope of what
25:38
you're trying to accomplish. And I think the auto GPT example
25:41
is actually a good one because let's be real, like a lot of the
25:46
tools that are surrounding agentic development today are
25:49
designed for for code native folks like full stop.
25:53
Yeah, that is not only because the the industry and I think the
25:58
applications in the field are moving very quickly.
26:01
And it's actually quite, it's much faster to iterate when
26:04
you're just working in a code native environment to try and
26:06
figure out like, OK, how do you land something with these new
26:09
capabilities that are coming off the shelf when people are
26:11
constantly innovating, right. It does not really permit
26:17
experimentation very well. OK.
26:20
And that's why and and this is by the way, this is not unique
26:25
to this is not unique to agents, OK, Her.
26:28
It wasn't that long ago that people were just training models
26:31
everywhere off the shelf and they were like, yeah, you know,
26:35
here you go. Like it's a paper.
26:37
Here's the code. Like, you know what?
26:40
Look at what he did. Look at what I did in this
26:42
paper. I did that, you know, and people
26:45
are like, I have no idea how to reproduce this.
26:48
This makes no sense. This code is spaghetti and I
26:51
can't stitch like the variables naming doesn't even make any
26:55
sense. I can't, I can't interpret it,
26:58
let alone contribute, let alone reproduce it.
27:00
So it's like, OK, you know, and then Hugging Face came along and
27:04
now they have like a whole way to serve these models within
27:07
friends, to test them out in a playground.
27:10
So, you know, I think it's a very, very good question if
27:17
you're trying to evaluate wands, you know, and you don't know if
27:22
it's the wand or maybe if it maybe it's the wizard, right?
27:25
Yeah. Don't go for, you know, I'm
27:30
going to go deep here. Don't go for Voldemort's twin
27:33
wand out-of-the-box, OK. Like there is nothing wrong with
27:38
the training wand. No one is asking you to bond,
27:42
you know, with like, you know, separatist Snapes, like, you
27:47
know, leftover wand, like figure out how to start with the tools
27:53
that are not bleeding edge, even the tools that are not leading
27:57
edge, right? Go with the most common
28:00
denominator. And there's nothing wrong with
28:03
that. OK.
28:04
And, and and if you're a code native person and you're having
28:07
trouble without a GPT, first of all, you're not alone.
28:11
I was struggling 6 months ago to set up a GPT myself too.
28:15
The documentation is good, but you know, sometimes you end up
28:18
in, in challenging spots, OK. And the environment variables
28:23
aren't exactly quickly documented since all places
28:25
anyway. It's fine, right?
28:27
And that's not to say that it's not a great library.
28:28
It's fantastic. I've used it before.
28:30
You know, that's an excellent library.
28:33
It has incredible contributors like I I really think that it's,
28:37
it's a great piece of software that's driven by the community.
28:40
It's open source and there are, as you mentioned, there are
28:43
several other libraries out there that are equally as high
28:46
quality. But the reality is that if you
28:48
were struggling with it, don't ask yourself, you know, to do
28:53
hard labour and explore cavern, you know, with without a
28:58
flashlight 'cause there are people that are doing that.
29:01
Figure out a way to pare back until you can come together,
29:07
cobble together a solution that looks pretty close to what you
29:12
want to accomplish. Even if that's, that's something
29:14
is like a no code solution. And that's a great way to start
29:16
like copilot studio as example. You know, there are, there are
29:19
tools off the shelf to do this stuff.
29:22
You know, don't, don't twist yourself into knots
29:24
unnecessarily. Now, you asked a very important
29:27
question, which is how do you evaluate the success or failure
29:33
given that you, you know, settled on a tool.
29:37
And the answer is quite simply, understand what success looks
29:41
like. Go back to, you know, what I was
29:44
talking about at the very beginning and be clear.
29:48
Are you able to accomplish with a clear mind and a clear intent
29:53
to the best of your ability with that tool, that success criteria
29:56
that you spelled out on paper? It's a very easy way of
30:00
understanding the problem. And you know, if you come all
30:04
the way back to like the no code thing that has everything, all
30:06
the bells and whistles out-of-the-box, you just can't
30:08
get there, Try the next step up, see if that gets you to where it
30:11
is. But be clear about that success
30:14
before you get started, because otherwise you're getting a loss
30:16
in this more DEXA tools. That's so good, that is, and it
30:22
I have to make the point that knowing what success looks like
30:29
is also not something that is unique to agents, right?
30:34
That is something and it's so easy to forget.
30:37
It's so easy to get wrapped up in all the cool stuff and the
30:41
shiny bells and whistles, and then next thing you know, you're
30:45
sitting there going like, oh, weren't we just supposed to do
30:48
that? I think if you if we may have
30:52
defined this a little better, we wouldn't have gone completely
30:57
off the rails. You know, as I as a product
31:02
manager, I, you know, I whatever you want to call me code fluid,
31:07
but don't get the code, you know, like just the, the person
31:09
that's in front of the customer, it says whatever, whatever you
31:11
want to call me, right? Like I subscribe to the school
31:15
of thought around customer development and the principles
31:19
of customer development and lean startup are really focused on
31:25
what is that granular piece of value that you're delivering.
31:32
And when you talk about thinking and, and architecting an agentic
31:36
system, like it or not, you are in fact creating a product.
31:45
And the reason why I say that is a product is designed to
31:51
accomplish a task through some combination of systems, whether
31:56
that is, you know, software that you're writing for yourself,
31:59
like whether it's like web software, you have like a
32:01
database, you have an application, you have whatever,
32:03
like other ancillary services, right?
32:06
The point is that people are using those services to
32:08
accomplish something. The difference with agents is
32:11
they're designed to accomplish those tasks autonomously.
32:16
You're giving them the tools to go accomplish something.
32:19
So in a way, each agent is itself a product and that gets,
32:26
you know, it can get, it can get difficult if you aren't thinking
32:33
about it with first principles in mind, right?
32:36
Because it can get impossibly complex.
32:38
If you put yourself in a situation where you say, I'm
32:42
going to have an agent for literally every single task and
32:44
they're all going to work together and they're going to
32:45
have a hierarchy and you're going to have a platter and you
32:47
know, we're going to put all this stuff together and it's
32:49
going to be beautiful. You know, and you're, and you're
32:51
thinking about it as like a, as a conductor and you know, you
32:55
have like the Symphony in your, this orchestra and you're just
32:58
like da da da da da da da da. Like, please keep it simple.
33:02
Please keep it simple, Please keep it simple.
33:06
You know, I, I will say that the, the, the best systems,
33:11
whether they are stitched out of AI or whether it's stitched out
33:14
of software, solve problems simply.
33:23
And if you can manage to get an agent to solve a problem, simply
33:30
you want. Yeah, yeah.
33:34
And it, it's specially exacerbated because if it can do
33:38
something in one hop that it in other times has taken 5 hops to
33:45
do, you're going to get more reliability.
33:48
You're going to get, it's going to be faster, everything is
33:50
going to be better because it is simple.
33:52
Right, right. And, and I think, you know, to
33:56
make it accessible to folks because I do think actually that
34:01
this is, and if you'll excuse me, Dimitrios, this is where I
34:05
kind of go a bit hyperbolic, but I, I do think, yeah.
34:10
Oh, indeed. I do think that agentic
34:14
workflows are going to be fundamentally transformative to
34:19
software at some level. Like Steve Jobs when he said the
34:25
good technology is indistinguishable from magic,
34:29
right? I think one of the things that
34:32
has happened, particularly with generative AI is that when we
34:36
first started interacting with Jet GPT, I'm talking about like
34:39
the layperson, right? Like just anyone at the show, we
34:43
immediately jump to like, the first thing you do is you ask
34:46
like a dumb question. What's you know, what do you
34:49
think about, you know, Abraham Lincoln's, you know, beard
34:53
speech or something, whatever, what theatre?
34:55
Like you ask if something dumb, just test how it goes and it
34:59
gives you this educated genuine response and you seize it and
35:05
you're like wait. That was pretty good, yeah.
35:11
The next thing that you do is you leap to what can this do for
35:18
me? And it wasn't until the
35:20
introduction really, in my opinion, of function calls, of
35:24
tools of things that you can offer to the agent, right?
35:26
Like access to the web, access to a database, access to
35:30
documentation, the ability to interact with an API, the
35:33
ability to perform inference of another model and obtain that
35:36
and use it for the purpose of predicting something else,
35:39
right? Like these are all capabilities
35:41
that are inherent in agents. It wasn't until we got to the
35:44
point where we can create that action space for these
35:50
transformer, you know, architecture, generative AI
35:53
models, these large language models that we were able to make
35:57
the leap from what can it do for me to how can I get it to do it
36:02
for me? And, and, and that is I think
36:06
where a gentic AI has such promise.
36:12
But you know, I'm going to put this caveat that it is very
36:17
early. That doesn't mean that you
36:18
shouldn't educate yourself. It doesn't mean you shouldn't
36:20
try, right? Like just because the iPhone was
36:25
like the first model doesn't mean it didn't have its hiccups.
36:28
You know, like I remember working with the iPhone press
36:31
and I was like, wow, this is magic.
36:32
This is really cool. I was like, you know, I still
36:35
don't understand why, why? Why can't I, like, I can't
36:39
figure out how to get the calling to work while I'm
36:41
texting somebody and like I, I just want to text, you know, It
36:44
had its flaws, OK? That's why software is
36:47
iterative, right? We are early in the agentic
36:51
software development life cycle. Give yourself and give the tools
36:57
that are offered in its stage of maturity some grace.
37:00
But don't, you know, try it and then be like, oh, it's too
37:05
early. I can't make sense of it, you
37:08
know, stay persistent because that piece of magic that
37:11
happened after you first, you know, interacted with the lol.
37:16
It's there. You just have to dig a little
37:18
bit and work on it. Yeah, you mentioned support
37:23
agents. I think that's a great use case
37:26
for agents. And there's some companies that
37:29
are just absolutely blowing up right now that are doing that
37:33
type of thing for agent workflows.
37:37
What are some other work flows that you've been seeing and you
37:40
appreciate? You know what?
37:42
I I'm gonna, I'm gonna, I'm gonna go rapid fire here.
37:45
OK. Oh, nice.
37:46
There you got. There's there are so many of
37:50
them. So many of them.
37:52
OK, so, all right, here we go. You ready?
37:55
I'm just gonna start going off the top of my head.
37:56
Hit me. Hit me.
37:58
All right, so look, you know you can fraud detection in finance,
38:03
right? Identifying suspicious
38:05
transactions and stopping them in their tracks and, and
38:10
contacting the user on that behalf.
38:12
Like intercept the fraud at that moment, right?
38:16
Supply chain optimization, right?
38:19
Bin packing is an age-old problem.
38:21
Everybody deals with it. It doesn't matter what your
38:24
industry you're in. OK.
38:25
And but in supply chain, it's so much more acute because there's
38:29
always real time information that is throwing things awry and
38:33
being able to act reflexively is the name of the game.
38:36
OK, patient monitoring. OK, let's say, you know, I'm
38:42
just, I'm going all over the place here because there's so
38:44
many applications. Like let's say you have an EHR,
38:47
you know, if you know anything about healthcare, there's all
38:50
kinds of information and telemetry in that EHR.
38:52
There's charts, there's doctor's notes, there's intake forms,
38:57
there's patient history, right? Like if there is an issue, why
39:02
are we waiting until somebody recognises there is an issue?
39:05
Why not have proactive monitoring to see if any of that
39:08
data flags to see if there should be a follow up?
39:13
Totally asynchronous, but a huge opportunity for agents.
39:18
I'll give you another one Offline, 'cause people always
39:20
think about online, They never think about the offline
39:22
applications of agents. I got a farm, right?
39:27
I've got sensors, I got telemetry in that farm.
39:31
What am I doing with those sensors?
39:32
What am I doing with that telemetry?
39:34
I'm doing nothing with it because I have to monitor it.
39:37
I have to sit on it and I have to figure out where I want.
39:40
Like, this is ridiculous. What's the fertiliser contact?
39:42
What's the nitrogen content of the soil?
39:45
What am I supposed to do with that?
39:47
No, make it actionable. Hand the agency over to an agent
39:51
that can turn on or off the water.
39:54
You know, apply nitrogen through kind of some kind of a drone or
39:58
some kind of a distribution system.
40:00
And see that that's. Cool.
40:01
I'll give you another one that I really love.
40:03
Here's another one I love. OK?
40:06
Personal shopping assistance. Take a snapshot of your closet.
40:12
Take a snap. Look, forget your closet.
40:15
Load up your social media, OK? Connect your Instagram account.
40:20
Inspirational. Yeah.
40:22
Yeah, totally. Whatever.
40:24
OK, get get your get your LLM some idea your multimodal LLM
40:29
some idea what you're wearing. Have it scour the web for your
40:33
brands and recommend you the clothes.
40:36
Why are you looking for the clothes?
40:38
Is it in you look this and the sales OK, for yeah, forget about
40:42
the clothes. What about the sales, right.
40:45
You don't have a personal shopper.
40:46
Why? You know, nobody's saying don't
40:49
go into the store, but don't you wanna know?
40:52
Like, why are like, you know what our alternative here is to
40:54
be trios. It's RSS feeds.
40:59
It's RSS feeds. That's the that's the closest
41:02
approximation you have. People are parsing through
41:06
emails and pamphlets to figure out like, how, how do I get the
41:11
best deal with the thing that I like?
41:13
Why are you doing that? Don't do it.
41:18
That is such a good one. I, I actually saw the other day
41:20
somebody created like a Tinder, but for clothing items.
41:25
And you could swipe right and like left.
41:27
And so if you swipe right, it saves it or it will take you
41:31
automatically to that. But it is, that's novel.
41:35
That's not your shopping assistant that can say, it would
41:38
be even cooler if you could say, hey, I just got you 5 shirts
41:41
because you're thinking about getting a new shirt.
41:44
These are all on sale. You've, you've showed interest
41:48
in them in some way, shape or form, or you have one of them in
41:51
your closet. You have been watching videos on
41:54
TikTok about these types of shirts, but going back to there
41:58
is something that I, I wanted to point out with those first two
42:01
use cases that you mentioned, which is that they're very real
42:05
time. The fraud and the supply chain
42:07
are very real time. And in my mind, the use of LLM
42:16
in real time never really went together because it is so slow,
42:21
right? And so, like a fraud, you can't
42:24
be sitting around for two minutes waiting to see if your
42:26
credit card passes. So there's the I, I'm actually
42:31
really glad you brought this point up.
42:32
To be honest. I think that there is a huge
42:39
investment opportunity that has yet to be fully realised and
42:45
movement around small language models.
42:49
I think we've looked at LMS as being a generalised set of
42:53
capabilities that offer, you know, a wide variety of
42:57
possibilities. But I think that, you know, as
43:02
the maturity life cycle of, you know, as an example, do you know
43:08
how many possible computer vision models, specifically
43:11
models have been trained? I mean, I innumerable like here
43:18
we are talking about like Lava 3.1, Lava 3.2, you know, Claude
43:24
3.5, you know, like get down to a dealership and get, you know,
43:30
will, you know, will 7.53, you know, it's like, look, the the
43:38
the point here is that the accessibility that small
43:44
language models represent for latency and inference and said
43:49
plainly, right, the smaller you get, the less expensive it is,
43:54
the more accurate it potentially is and the more powerful it is.
43:57
OK, and the fast and the faster it is fundamentally and the
44:01
faster it is, the cheaper it is, the faster it is.
44:03
There's also a parallel movement that's been going on around, you
44:10
know, quantization and parameter efficient fine tuning.
44:16
And I think that, you know, as we start exploring a lot of
44:21
these use cases, the inference cost, whether that is done in
44:27
terms of tokens, whether that's done in terms of latency,
44:30
whether it's done in terms of, you know, RAM size, right?
44:35
The actual compute resources that are required for inference.
44:38
You know, we are going to continue to see that decrease
44:41
and decline. It it this is the part where I
44:44
have to say that, you know, this is kind of on more of the
44:48
bleeding edge in terms of the return on investment.
44:52
But the, the key here, the important thing here is if you
44:56
can get yourself and your application to the point where,
45:00
you know, inference speed and cost is the problem you already,
45:07
because you've already solved the problem of how do you
45:10
achieve your goals? Have you delivered success?
45:14
And there are tools out there that will allow you to do it.
45:17
And even if you have to go and hire data scientists to figure
45:19
out like how do I train an SLM on this data set?
45:22
Or how do I provide, like, you know, a graph rag interface that
45:26
will make this more efficient and, and quicker.
45:29
You know, how do I reduce the amount of resources that's
45:32
associated with inference because I'm getting too much,
45:34
you know, consumption downstream on my API?
45:37
Like, these are great problems to have, right?
45:40
Especially if you deliver in real time.
45:42
Like if you can prove that the value of supply chain
45:46
optimizations happens in a time series, right?
45:51
If you simulate the outcomes and you can showcase that you're
45:54
more efficiently allocating resources in your supply chain.
45:57
And you can do that effectively. And the only barrier to entry is
46:01
that you need to make that faster.
46:03
You are in a very good spot, my friend.
46:06
You're in a very good spot. Yeah, that's that is a great
46:11
point. The speed is the last thing.
46:16
There's so many other questions that you need to tackle or so
46:20
many other obstacles we could say, and hurdles that you need
46:23
to get through. By the time you get to speed as
46:27
the hurdle, you're looking good. And so instead of jumping, I
46:32
guess I, I kind of jumped to the end and say, whoa, is it ever
46:36
going to be that fast? I, yeah, I understand there's
46:39
distilling. I understand small language
46:41
models, but I also see the the side of if I'm doing fraud and
46:46
it has to be quick. I do think there is a world,
46:50
though, a little tangent where you get these workflows that are
46:56
using a combination of small language models plus just
47:00
traditional machine learning, plus you've got these workflows
47:05
that are are plugging in. And again, going back to that
47:07
orchestration, each step on the orchestration, whether
47:11
synchronous or asynchronous, it doesn't need to be every time
47:16
there's a step on the orchestration or a gentic
47:19
workflow. It has to be LLM call or just
47:23
small language model call. It can be a regular regex.
47:27
It can be whatever you want it to be, right?
47:30
And so whatever you find as the most optimised.
47:33
And also then going to that, because I know people when they
47:37
go into production, one of the very difficult parts of, again,
47:41
walking that tightrope of the trade-offs is, yeah, we can get
47:46
it fast, but we can't get it reliable.
47:49
Yeah, can once you start trying to shave off time, then you
47:53
start seeing the accuracy go down.
47:55
And that's a classic problem that is as old as machine
48:00
learning. If we're going back to these,
48:01
the 50s, as you mentioned, since agents been around, that's
48:05
that's kind of been the name of the game.
48:07
Yep. And I think, look, here's the
48:10
here's the deal. You're not alone in this.
48:13
Like it's human nature to confront something that is new
48:19
and make a justification why that new thing will fail.
48:22
And whether that is intentional or unintentional, whether that
48:25
is justified or not justified, I think it it, it is it really
48:29
behoves us to kind of see past those initial objections and
48:33
figure out, is the value worth it?
48:37
And quite frankly, you know, am I just, am I just saying it's
48:42
not going to work because I don't really know how it would
48:45
work? Or am I just saying?
48:49
Lean on the braid. Yeah, or, or I, I just want to
48:52
discount the the potential, right?
48:55
It's OK. Nobody's going to fault you for
48:59
calling out what is what are quite frankly the main
49:04
impediments to rolling this thing out at, you know, mega
49:10
cluster scale, right? Nobody's, you know, if you, if
49:14
you go to somebody on the street and they're like, you know, why
49:18
don't I have chachi PT everywhere I look?
49:20
Well, yeah, I mean, it's a problem.
49:22
I mean, there's other stuff, there's work you got to do, but
49:25
the, the truth is that, you know, all of I, I wouldn't say
49:30
all like they're an, there's an incredible concentration of
49:35
minds that is focused in trying to figure out alternatives to
49:43
these objections. And just because if they're
49:48
working on it doesn't mean they're going to solve it, but
49:50
it also doesn't mean that you can outright cast it out.
49:54
Yeah, that put a very not scary but but fun picture in my head
50:01
of, yeah, like these brilliant folks that with their
50:05
candlelight. Just working on figuring out how
50:08
to make this, how to get this working.
50:11
And it's like this army of smart people and then me, the
50:14
naysayer, being like, no, that shit ain't going to work.
50:17
Yeah, right. It's like.
50:18
Dude, you know how many smart people are working on this
50:20
problem? Yeah.
50:22
But it's, it's fine. It's fine.
50:24
You know, like we've seen this happen everywhere.
50:27
You know, people said the same people said the same thing
50:31
about, about the web. I'd be totally honest.
50:37
Look where we are today. Yeah, yeah.
50:40
We're having a call. You're halfway across the world
50:42
and we're having a call, video call nonetheless, on the web.
50:47
They're in the browser. In the browser.