Latest / Agentic Conversations (formally mlops.community) / Machine Learning, AI Agents, and Autonomy // Egor Kraev // #282

Transcript

Machine Learning, AI Agents, and Autonomy // Egor Kraev // #282 · Agentic Conversations (formally mlops.community)

0:00
So hi, I'm Iego Kraev. I am now principal AI scientist
0:06
and wise until recently heading up and building up its AI team.
0:11
Parallel to that, I'm fortunate to also be able to work on my
0:15
startup on causal inference in marketing code name Causaltun I
0:23
I instead of coffee I prefer green tea.
0:26
Things like long gene, for example, There is, you know,
0:30
there are certain things, there are certain things, but once you
0:32
know what how the good stuff tastes, you cannot really drink
0:35
the best stuff anymore. Like wine, whiskey and green tea
0:38
is certain of them. Welcome back to the ML Alps
0:42
Community Podcast. I'm your host Dimitrios, and
0:44
today we got into the traditional ML, the AI world of
0:50
things, and also the AI agents world of things.
0:52
And Igor has built packages and open source products for each
0:57
one of those. It was really cool to dig into
1:00
what exactly he's done with the different ones, be it fraud
1:05
detection or segmentation and AB testing in emails or LLMS as
1:11
being just one part of your DAG and how it's, it's really useful
1:16
to think of an LLM as taking unstructured, messy data and
1:22
turning it into structured data. And lastly, the framework Motley
1:27
crew that he created around AI agents so that you can use
1:31
various AI agent tools or frameworks if you want to.
1:37
You don't have to be locked into just lane chain or lane graph or
1:41
Lum index or crew. AI Monthly Crew allows you to
1:45
have a monthly crew of agents and leverage what's best about
1:49
each one of those. Let's jump into this
1:54
conversation now. And for those who are on
1:57
Spotify, I've got a nice little song recommendation for you
2:01
coming right up, which is from my new favourite playlist, Big
2:06
Desk Energy. And this song is called Smeds by
2:12
Steve Cardigan. That saxophone is just too good.
2:59
All right, folks, I'll catch you on the other side.
3:07
Have a very happy holidays and a wonderful new year.
3:15
Let's get into this conversation.
3:38
I think we spoke two or three times in the past year and we
3:42
didn't record any of those conversations.
3:45
And now I'm glad that we are finally recording this
3:48
conversation. So before we get into any of the
3:53
technical stuff which I want to talk about, because as I
3:56
mentioned many times when we spoke, I'm a happy user of Wise.
4:01
I love the product. I know you're leading all kinds
4:04
of AI initiatives at Wise. What's up with the pirate stuff?
4:09
That's what I really want to know.
4:11
I am one of the founders of the Swiss Pirate Party and.
4:17
What does that even mean? Well, it means whatever people
4:21
make it mean, of course. For me, what it means is that I
4:25
think the balance between copyright and public access is
4:29
way out of is way out of line in pretty much all Western
4:34
countries or all countries I'm aware of really.
4:38
Because having being able to control anything, any
4:41
information that you created is not a God-given, right?
4:44
Right. It's a monopoly created by
4:46
government and therefore that monopoly must serve the public
4:49
good. And because the monopoly always
4:51
destroys value. So it's only worthwhile if it
4:54
also adds value. And now, for example, the
4:56
thriving open source products which we all know from Linux to
5:00
countless things, show that it's not necessary in all cases to
5:06
have copyright protection for good things to come into to come
5:09
into being. Sometimes it is, sometimes it
5:11
isn't. And So what?
5:12
For me, what the Pirate Party is striving to do is to shift the
5:16
balance a little bit. Wow, OK so I was way off base
5:21
when I thought it was like you all were dressing up as pirates,
5:25
maybe outside of Halloween and having parties.
5:29
When I saw that written I thought, oh man, he likes to
5:34
wear patches over his eyes and. No, it's the other kind of
5:37
pirates. It's a classic example of
5:39
concept. Hijack.
5:41
Yeah, great. You did well.
5:45
You had me. I'm very misdirected.
5:48
So the other piece that is cool to talk about before we jump
5:53
into Wise and everything you do in there is the work that you've
5:56
done over the years before you hit Wise.
5:58
And I know you were in Africa, you were in Ghana for a little
6:02
bit, right? And yeah, can you explain what
6:06
you were doing there? Oh that was a wonderful story.
6:10
I was a student in the US and being a young passionate
6:15
student, I got involved in the anti globalisation protests
6:18
against the IMF and the World Bank, doing bad things or what
6:24
the organisers said about things in all sorts of developing
6:26
countries. And then I was part of those
6:31
protests. It was all very nice, very
6:33
civilised. In fact I was reminded of
6:36
umberta Echo's descriptions in the Name of the Rose about the
6:40
heretics. It's like a carnival.
6:42
It was this wonderful, wonderful thing and that as part of this,
6:44
some of us were invited to the IMF where the people from the
6:48
IMF were explaining to us what it is actually do and they're
6:51
not actually evil and all that. And after that, just that very
6:56
night, there was a party at one of the nonprofits that I was
7:00
that I was hanging out with. And there I was telling the lady
7:04
who was organising the party and who had a nonprofit.
7:08
Like I was telling her, like this IMF guy took such a
7:10
horrible, such a horrible job of explaining what they did.
7:14
Like, I could have done a better job of explaining what they do,
7:17
clearer and shorter and anything.
7:20
And then she said to me, well, there's somebody I'd like you to
7:22
meet. And then she introduced me to
7:24
Charles Abu Grey, who was a big guy in the nonprofit scene.
7:28
So he had a start up in Ghana doing all sorts of public good
7:32
stuff. And then he invited me over for
7:35
a week, first of all. And then we got along.
7:38
And then I ended up spending at least half of my PhD there doing
7:42
all kinds of Economic Research for that nonprofit.
7:47
What kind of Economic Research was it?
7:50
It was quite mundane actually. So relationships between
7:55
inflation and income distribution, inequality, that
7:59
sort of thing. But to what made me then move
8:03
on, change, change careers is the realisation that it doesn't
8:06
really matter what research I was doing because the name of
8:10
the game is just by having somebody who does any kind of
8:14
credible research move the move the nonprofit to a different
8:18
league. So they were invited to
8:19
different tables and could take part in different conversations
8:22
on the back of that. But the content itself was
8:25
irrelevant and once or largely irrelevant as long as it was
8:28
largely pointing in the right direction that the nonprofit was
8:31
aligning with. And once I realised that I and
8:35
also got tired of being poor, which is part of working as a
8:38
nonprofit in Africa, I changed. I changed careers and went to an
8:43
investment bank. So you went on the exact
8:48
opposite side. You're like, you know, that guy
8:50
from the IMF, He actually that stuff they're doing wasn't so
8:54
bad. Maybe I should get try my hand
8:56
at finance. Yeah.
8:59
You know, if you can't beat and join them.
9:01
Yeah, incredible. Well, I want to say a few things
9:05
that I've jotted down from our conversations and I would love
9:08
for you to elaborate on them because every time that I talk
9:12
to you, I feel like you have a tonne of hot takes and you are
9:18
knee deep in so many different areas of AI and ML, from
9:24
traditional ML all the way to LLMS and AI, quote unquote, all
9:30
the way to AI agents. And so one thing that you said
9:35
to me that stuck with me was AI is a bridge from unstructured to
9:42
structured. Can you explain what you mean by
9:44
that? Well I would rather say that one
9:48
of the biggest trolls for large language models is being a
9:52
bridge between unstructured and structured data.
9:55
Because if you think about the way we did data science only two
9:58
years ago is you first had to make everything into a vector or
10:03
a matrix and then you would begin working with it.
10:07
So when do you when do you do topic decomposition?
10:10
Then topics were just areas in vector space really.
10:14
And then you had a hell of a job explaining what they even meant
10:17
and you have no hope in hell for normal human to modify them.
10:20
Whereas now if you do topic classification then topics
10:24
descriptions are the topics because the LM going to work
10:27
with the draw text directly and at least 3/4 of production
10:32
implications for lamps that I've seen are just the LLM converting
10:37
incoming fluffy language data into something structured.
10:43
Is that a customer complaint? What is the what is the rate on
10:48
this contract? The late charges percentage
10:51
rate. Input goes a contract, out goes
10:54
a number, and the vast and the vast majority of the
10:57
applications that I've actually seen are just that.
11:03
So taking a lot of messy data of the unstructured and then trying
11:09
to figure out some kind of a system so that that's the input
11:13
and the output is something structured.
11:15
Exactly. And in fact, I think the vast
11:19
majority of use cases for LMS that actually work in production
11:24
are are not just in comes the magic LM, in comes the magic
11:28
agent and does everything, but rather you take the LM as an
11:32
additional Lego brick in addition to all the other bricks
11:34
you have and you combine them. And that's how we get out the
11:38
value. OK, Now do you think that it's
11:44
easier to explain or champion for AI now that it feels like
11:50
it's more simple to grasp like that then when you were doing ML
11:55
two years ago and you had to try to tell someone, oh, well, we're
11:59
going to vectorize this and then we're going to matrices and all
12:03
of that jargon. Hopefully you weren't doing that
12:05
when you were explaining it to leadership or, or presenting
12:08
different use cases. But now you don't really have to
12:12
do that, right? You can speak at a different
12:16
level. Maybe, but really the problem is
12:23
never the technology or even being able to explain the
12:26
technology, because if you know what you're doing, then you're
12:30
able to explain it. But there seems just they always
12:34
the the biggest blockers are always of two kinds, a
12:37
organisational structure. But this can be broken down with
12:42
sufficient will from above at least.
12:45
And the other one is the invisible walls in people's
12:47
heads. Because there's a fun thing I've
12:50
observed with a variety of technologies, from bringing FX
12:54
swaps to Wise Treasury to now LMS in customer support to a
12:59
bunch of others, is that even if you have a team and then you
13:03
have a which has an existing workflow that kind of works for
13:06
them and you bring a new technology to them, which
13:08
clearly adds value, but they have never, which they have
13:11
never seen before. And then from the moment you
13:14
start trying to educate them about it to the moment it
13:18
becomes just another thing in common place.
13:21
Minimum of two years. Wow.
13:23
It just doesn't. It doesn't matter how hard you
13:25
try. It just takes time for people's
13:28
heads to adjust around this thing, along with all the other
13:31
stuff, all the way around all the other problems I have.
13:33
It's two years. Yeah.
13:35
And to build those habits, to build those muscles of I'm going
13:38
to use this instead of my traditional workflow.
13:41
That makes sense. So if you budget for two years,
13:44
then you set your expectations at a realistic level and if it
13:48
happens before then maybe you get lucky.
13:51
Yes, and generally, generally it won't.
13:54
Like you could have the prototype out in two weeks, but
13:57
for people to actually adjust to the fact that exists.
14:00
This is how it works, that she's good for you.
14:02
It's not scary. Two years.
14:04
Yeah, and if it's more than that, it's time to start looking
14:08
for a new job. Perhaps, Perhaps thankfully, you
14:12
know, thankfully wise is actually one of the more in gaol
14:15
places. So it's I have.
14:18
I have never had good ideas blocked here.
14:21
Well, there's plenty of ways that you're using ML and AI wise
14:27
from the traditional fraud detection because it is a
14:30
financial services company or it's a financial fintech
14:34
company, I think is what you would categorise it as, right?
14:37
And for those who don't know, it makes it really easy.
14:41
And the reason that I love using it is because I can have money
14:44
in the US and I live in Europe. And so it's really easy for me
14:48
to move that money around and not have to pay exorbitant fees
14:55
from traditional banks. And I can just juxtapose the
14:59
differences of when I would come to Europe when I was like 19 and
15:03
20. And so why is kind of cut that
15:08
all out and made it really easy? So coming back to the AI and ML
15:13
that you're using at wise, since it is finance, I imagine there's
15:19
a lot of a fraud detection and then B maybe some are you doing
15:25
loans? I don't think you give loans do.
15:27
You no that we don't, but there is a lot of things.
15:32
So absolutely, of course, fraud, detention, anti money
15:35
laundering, the whole wide, wide area of think frame is probably
15:40
the oldest because it's so clearly beneficial.
15:43
And that's classic ML, so wide table of data, extra boost to
15:48
high primes, tuning the whole thing.
15:50
And, and of course, and that's also the area that our PR
15:53
department is not very fond of us telling my details about.
15:58
And then there's also understandable for good, for
16:01
good reasons. The other one is treasury and
16:05
treasury is what you call a trading desk in in banks because
16:09
the flows, the people want us to transfer and always balance.
16:13
And so you have to go to the interbank market in one way or
16:16
the other and source the necessary currency and then
16:20
manage the risk of holding currencies.
16:22
Because one way of looking at wise is actually as a market
16:26
maker to the masses. You always will always give a
16:29
bid and ask price for any currency pair.
16:32
So effectively we are a kind of market maker.
16:34
But a very unusual 1 because we market maker to the masses
16:38
rather than to other big financial institutions and we
16:40
try to make you pass press as tight as possible as opposed to
16:44
the banks who try to see how much they can get away with.
16:48
But so Treasury is a lot of machine learning, trading
16:53
estimation of flows to make sure we have cash in place.
16:57
For example, did you know that if you try to withdraw money in
17:01
Wise in Sri Lanka or places like with the currency control
17:06
places, actually it's not. We Wise has to send the dollars
17:11
to our partner bank the day before so they can arrive
17:13
overnight and wait there safely at the partner bank.
17:16
So then when somebody wants to withdraw money, Wise can ask the
17:18
partner bank to please charge us the dollars and give you the
17:21
local currency. So there is a lot going under
17:25
the hood for this quasi instance experience.
17:29
And then my final big area of course, or my final favourite
17:32
area, of course, it's not, not by far not the only one is
17:36
marketing and causal in France and all those fun and games.
17:40
Yeah, well, and you did kind of mention before too support,
17:45
right? Absolutely.
17:48
I think support is comparatively young because before LLMS
17:53
happened that it was quite hard to do because so much of the
17:57
data is tax based and but now we have a great data science team
18:01
in place there and they were to have some things in production
18:04
and more is on the way. So another thing that you said
18:09
that stuck with me and I want you to elaborate on is LLMS or
18:14
AI shouldn't be necessarily looked at as the solution.
18:18
It should be looked at more as one step to get to the solution
18:23
almost. And the way that I understood it
18:25
is we should just look at it as another step in a DAG.
18:32
Yeah, 100%. I also, I, I find it very silly
18:36
when people ask KGPT to add two and two and JGPT sells them as
18:41
five and they're like, oh, AI has failed.
18:45
It's the whole thing is like, you should think about this as
18:48
one big Lego set and now you have a couple of extra blocks
18:53
that you couldn't do before. Maybe the blocks can blink or
18:55
emit sounds or whatever it is they do, but there's just one
19:00
more addition to your Lego set and it'll it'll be sealed and
19:06
then it reads. It gets its power from being
19:08
combined with all the others. And I'm sure it won't be the
19:10
last cool thing either, because I still remember that when RNNS
19:15
arrived 10 years ago, it was exactly the same kind of hype
19:18
wave or air and ends and neural networks will solve machine
19:21
learning for us. And then for, and that kind of
19:23
settles down for for, you know, eight years or so, there's
19:26
nothing. And there's another big thing
19:27
comes along. I'm sure that'll be another one
19:31
now Which? Ways are you seeing both LMS or
19:35
foundational models plus other traditional ML or just regular
19:43
heuristics being used together? Well, so the most, the most
19:51
obvious 1 is just LLMS being used in old school pipelines.
19:57
Or as I mentioned, LLMS just transform data.
20:00
For example, they give you a score of four.
20:04
Does this, does this customer e-mail look like a complain,
20:08
like a complaint based on the text?
20:09
What is the likelihood? Oh, and then you add that score
20:13
to a bunch of other data points you might have and do old school
20:15
machine learning to classify the e-mail.
20:18
And that is at this point this is the easiest thing to do, the
20:21
most controllable, the safest and so also the most production
20:24
radio for the other ones. I guess the fun part about LLMS
20:33
is that they humanise machine learning in so many ways.
20:36
So they they make this interface between they make this interface
20:41
between humans and machine learning fluid.
20:45
For example, in customer support bots, right before you had a,
20:48
you had this text block and then you had to classify it.
20:51
What is the customer asking about is you know, it's called
20:53
modal vector spaces, yada yada. Now you can just ask an LLM.
20:58
Does this in from message contain enough information to
21:01
understand what the customer wants?
21:02
If not, what you get what? What else should I ask the
21:05
customer? And that's little prompt.
21:07
And then you can ask the customer For more information
21:10
and they'll give it to you so that you unlock this whole
21:13
interactive potential that just wasn't there.
21:18
And I want to talk about causal inference technology and what
21:24
you're doing there because you said marketing and all of that
21:27
fun stuff is really one of your passions.
21:31
Give me the lay of the land on what you're doing, how you're
21:34
doing it and what it looks like with MLA.
21:37
I, I know it's, it's AB testing, right, But what else is going on
21:41
there? Well.
21:42
It's not just AB testing, but the trick is that the trick is
21:49
estimating causal impacts, and that is hard.
21:53
It's not like regular machine learning because in regular
21:57
machine learning, for example, if you want to predict how much
22:01
a customer will buy in the next month, then after a month you
22:05
can see how much have they bought.
22:06
So you have an observed true value, and then you can rate
22:10
your prediction or the multiple prediction variance, you can
22:12
rate which one was closest when you send.
22:16
When you choose to send a customer e-mail Ala versus
22:19
e-mail B, you really have no way of measuring that impact
22:23
directly because you cannot send only e-mail A and only e-mail B
22:26
to the same customer and compare.
22:29
So that's quite hard. And then, unsurprisingly, people
22:32
have built models specifically for this case, causal inference
22:36
models. So for example, Microsoft's
22:38
economic library is wonderful, but then again, they have half a
22:43
dozen different models, each with its own hyperparameter
22:45
universe, and absolutely no guidance about which one to use.
22:50
So what we've done twice under my guidance is to find a way to
22:57
score these. So even though you can't observe
23:00
individual impacts, turns out if you have a whole population like
23:05
an AB test, you can score these models out of sample.
23:09
And once you can score out of sample, you can do model
23:12
selection, hyperparameter tuning, all those wonderful
23:14
autumnal things. And so now you actually have,
23:18
even though you can't observe them directly, you have a
23:20
verified and selected estimate of impact for every single
23:24
customer. And then, so now you can, once
23:27
you have that estimate, you can do fun things, right?
23:31
You can do targeting first of all, so you can send to the
23:34
customer the e-mail is most likely to have them do the
23:37
action you want them to do, like click on that link.
23:41
You can also do segmentation. You can see which parts, once
23:44
you have that impact a customer level, you can now segment much
23:47
cleaner because how do people do AB test segmentation now?
23:51
They slice the whole AB test sample in little chunks and try
23:55
to see if there's significance. But that's very, very noisy.
23:57
That doesn't work this way. You can.
24:00
How many customers or people do you need to have on a list in
24:06
order for this to actually have like statistical relevance?
24:12
So certainly as many as you have for a regular AB test is enough.
24:20
Usually I expect it to be even smaller because now instead of
24:25
treating in an AB test the whole customer variability, you treat
24:30
like noise, right? You average over it.
24:32
All you want is an average. Whereas with all these models,
24:36
you first model customer variable, customer natural
24:38
variability based on the customer features, what you know
24:41
about them. Because after this he can, you
24:43
can only model the impact on top of this.
24:46
So now you actually treat customer variability in in the
24:49
customer behaviour signal. And so this is also, so we
24:53
haven't tested that extensively, but I expect we actually need
24:56
smaller sample sizes than for a Glaber test.
25:00
And is this what you're doing with Wise Pizza or is is why is
25:03
pizza a little different? Why is pizza is related?
25:08
So why is pizza is there for for finding fun segments?
25:12
It started with growth analysis. Supposing you have this data set
25:16
with customer segments, regions, the device they use, region
25:20
they're in customer from from currency to currency product
25:24
they use to any number of dimensions you might want to
25:26
have for your customers. And then so you have like a
25:29
million micro segments and now you want to find out my overall
25:34
say error revenues per customer went down by 2% or went up by
25:40
10% or from 1/4 to the next. And you want to find out which
25:45
simple explainable segments were driving this and that's advice
25:49
pizza does or Rick, my growth rate went down from went down by
25:54
1% from 1/4 to the next. What were the main customer
25:59
segments explained in simple terms in terms of those
26:02
dimensions I gave you that drove it?
26:04
And you could also apply it in causal tune results.
26:06
So you could also apply it on causal inference results, but
26:08
you don't have. To do you always find segments
26:14
that match up and that you understand, oh, it's, it's these
26:17
people with these different features that drove whatever the
26:22
question was whether it's an increase or decrease in revenue
26:26
and or is it sometimes that it just is a little bit scattered
26:31
like or or very weighted towards one or two people that don't
26:37
really have anything in common, I guess.
26:41
Well, in twice we're fortunate to in this sense fortunate to be
26:47
AB to C company. So we have large, large customer
26:50
bases and so, so sample size is not a problem.
26:55
And this thing will always find something, right?
26:58
Because that's machine learning. That's what machine learning
27:00
models are built to do. You tell them to find something,
27:03
they will find it. And then you could always just
27:06
split your sample in half and fit it on 1/2 and then look at
27:09
statistical significance on the other half.
27:11
And if it's there that it's really something.
27:16
Wow. That is so cool.
27:19
And how do you bring that? Because wow.
27:24
So let's say that you are answering some of these
27:28
questions and you're finding different segments that are
27:31
driving the reasons and you've split it in half and then you've
27:37
recognised, OK, this, this seems to be true, right?
27:41
Then what do you do with that information?
27:45
Well, that's really where humans step in.
27:49
In our case, we have So what the rest of the world calls data
27:53
scientists, the wires calls analysts, and what the rest of
27:58
what wires calls data scientists is more like an ML research
28:01
engineer in the rest of the world.
28:05
And so we have those wonderful analysts who then go in and dig
28:08
deep into those segments. And now they don't have to go
28:11
through little while wandering around in pivot tables looking
28:14
for something that's out of line.
28:15
They can clearly see these are the drivers.
28:17
OK, now let's look at, say, iPhone users in Asia.
28:20
What happened there? Or it's really the big tickets
28:24
that drive the change. So let's look at the big
28:27
tickets. What's going on there.
28:29
But that's I think also a bigger philosophical point.
28:32
I have exactly 0 fear the two to be replaced by algorithms.
28:39
In fact, the more you know, the more machine learning can do
28:43
things that are currently manual labour by data scientists, the
28:47
more data scientists can use them and therefore add even more
28:51
value. So the more labour replacing
28:53
stuff like AGPD happens, the more jobs that will be for data
28:56
scientists, not the other way around.
28:58
Fascinating. And so you're, you're digging
29:01
around in that data, you're trying to find insights and then
29:04
they're presented to leadership and hopefully you're whatever
29:10
creating a new campaign to try and target that specific segment
29:15
or you're running a a discount or you're doing something, some
29:18
kind of action is taken to help with that.
29:22
Exactly. And also, of course, it wouldn't
29:26
be 2024 if I wasn't currently working on an extended agenda
29:32
question of this, where you run this analysis, you come up with
29:35
those segments, and then you see whether you can source some more
29:38
data from wikis or chat or Slack channels or any kind of internal
29:43
information you have that would help explain it or be maybe be
29:46
relevant. You still never fully automate
29:49
it, but you could go a long way to.
29:52
But you would have an interactive tool where, in
29:54
dialogue with the machine, humans could do a much better
29:57
job of telling the whole story, not just the numbers.
30:01
So if I'm understanding you correctly, you're analysts are
30:06
digging through very specific segments of users who are
30:12
driving some question that you have.
30:16
And let's use this example of revenue went down 1% in the last
30:21
quarter. So let's figure out who are the
30:23
major reasons for that and you find out that it's iPhone users
30:29
in Asia or something along those lines.
30:31
Analysts are digging through that data and then they can
30:36
augment that data, that structured data with, oh, you
30:40
know what happened, We raised the prices in Asia for XYZ
30:46
transfers or we did something. And those analysts would not
30:50
know that unless they had the agent bringing them extra
30:54
context. Absolutely.
30:58
And this is not well. So I don't want to over promise
31:02
there's not something we have just yet.
31:03
It's something I'm actually working on right now.
31:06
So hopefully we'll have something rough out by end of
31:09
year. But that's a niche that I don't
31:12
really see occupied right now because like, let's let's ingest
31:17
all your corporate data and put a chatbot on top kind of
31:20
startups or a dime a dozen. Yeah, such a natural idea.
31:24
But combining this with the qualitative analysis to do
31:30
storytelling that's that's linked to the to what the data
31:34
tells you, that's not really something I've seen out there.
31:38
And so in a way, it's just it's plugging into all of your
31:42
internal docs, your internal messaging systems, and it's
31:46
trying to find relevant information to the data segment
31:51
or how is the, how are you interacting with the agent?
31:54
Is it through a chat bot? Yes, I think so.
31:58
It's just just, it's just a chat dialogue.
32:01
So again, this is now very much in the design stage again, but
32:05
it's very much just a dialogue. So you have the kind of report
32:09
with draughts and text that The thing is generating.
32:12
And then there is a chat panel to one side telling you where
32:15
you can say, oh, no, let's zoom into the segment or what's going
32:18
on here. Or they make changes report this
32:20
way because I think it has to be a dialogue because ultimately
32:25
humans, they know what story they want to tell.
32:30
And this has to be a story that's supported by the data.
32:32
But potentially there are many stories that could be supported
32:35
by the data and ultimately it's a human thing.
32:37
Of those stories, which are, which is the one you actually
32:40
want to focus on this time? Yeah, I also like the UX.
32:44
Sometimes when I'm playing around with different agent chat
32:48
bots of it will guess what question I want so I don't have
32:53
to think as hard. And for me it's a lot easier
32:56
just say, Oh yeah, let's see what's going on there.
32:59
Almost like when you click on a YouTube video because the title
33:03
entices you. So the agent comes up with like
33:07
4 questions that maybe you might want to know.
33:10
And I've seen this done where you ask an initial question and
33:14
then there's the follow up questions.
33:16
That's a pretty common theme that is in agents these days or
33:21
just in chat bots in general. And it, it feels like it would
33:26
be very cool if you're moving around in the data and you're
33:31
asking questions and then the agent can suggest things like,
33:35
oh, maybe you want to know about XYZ.
33:39
And so I like the way that you're going about that.
33:45
And it also is is very nice to augment the capabilities and
33:49
augment the tools that an analyst or a data scientist has
33:54
at their fingertips to be able to tell the story that they want
33:58
to tell. As you were saying, when you're
34:00
building out the agents, what have been things that have been
34:04
difficult? I am afraid it's the usual right
34:12
prerequisites, data quality and then convincing and then get it
34:18
working with engineering lead times for getting things into
34:21
production. So those are but because in chat
34:28
bot in its own is good as a toy. But for example, if you want a
34:32
chat bot to answer customer queries, you need to have a hard
34:38
but old school underlying taxonomy of what the possible
34:41
customer questions you could handle are.
34:44
And you have to distil that taxonomy and build it and expose
34:47
it to the agent. And then the actual Asian part
34:50
is easy. In fact, I think that's the
34:54
that's the general theme in the whole hype around agents.
34:57
Like it's kind of fun, but I think agents like using an agent
35:01
in itself will be kind of written very shortly, will be
35:05
kind of like using a database. Like, yeah, they're useful to
35:08
add value, but they're it's not a big deal.
35:11
It's like, oh, we have a database and we have a multi
35:14
database application. How wonderful it's, you know,
35:17
it's just a pattern and it's sort of it's sort of very hard
35:20
pattern. All the whole the stuff around
35:22
around you to make it work a hard agents as well.
35:25
So easy. The database is not for the
35:32
majority of people. It's not the most sexy of
35:35
technologies. It's kind of old.
35:37
Maybe it was back in the day, but now it is.
35:40
Like you said, it's just the database.
35:43
Let's the ultimate success, actually.
35:45
Yeah, yeah. So is that the maturity of where
35:50
it's at? So the thing that I, I'm
35:54
wondering around agents is how are you making sure that the
36:03
right data is not going to the wrong person or the wrong data
36:11
is not going to the wrong person?
36:13
I, I guess when it comes to for example, with Google Docs, you
36:18
have a very clear sharing protocol and I'm not worried
36:25
that somebody is going to be able to see all my Google Docs.
36:27
But with agents, I think it's really easy to not have that
36:32
role based access. And so the agent can now get
36:36
into any data that it wants to even in private Slack channels
36:40
or whatever. And then you're surfacing
36:43
information that that analysts maybe shouldn't have access to?
36:49
Well, that's an excellent point actually.
36:53
And to some well on on one level we don't really have that
36:57
problem yet because we don't have, we certainly don't have
37:01
customer facing agents which directly output all of them.
37:04
So all this thing is human in the loop, but and also internal
37:10
tools, most of these things, they provide draughts which
37:14
humans then can edit. But in the long run, actually
37:18
the problem you address is not that difficult because when you
37:21
the way you store things, you think about how these things
37:23
work like you ingest, you ingest snippets from everywhere and
37:30
then those snippets and then via some kind of rag.
37:34
And I could go on about rag for a day, probably will be giving
37:37
some talks and it's sue and stuff on.
37:41
But you retrieve some of the snippets that might look
37:43
relevant or that look like they might be relevant.
37:46
And then you give those snippets in the prompt.
37:48
They'll analogue with the question.
37:51
Now there is nothing easier than to attach to those snippets
37:54
metadata where we shall they came from and then philtre by
37:58
that. So that's actually not a very
38:02
hard problem, but it's an important one.
38:07
Oh, so you're doing it within the metadata of the different
38:11
chunks as opposed to in the database itself?
38:16
Oh. I'm not sure how else you would
38:19
do it because there are so many different systems and the
38:22
permissions are granular and entangled.
38:25
Almost all you could the best you can hope for is just is just
38:29
flag. OK this chunk friend, these are
38:32
the this chunk came from here. Is this person titled to the
38:35
source yes or no? Yeah, yeah.
38:38
Is this person in that Channel? If it's a private channel and if
38:42
not, then make sure not to include it exactly and.
38:46
Who could they read? Are they allowed to read this
38:48
wiki page? And so on and so forth.
38:51
Yeah, OK, so now tell me about Mötley Crüe.
38:58
Mötley Crüe is fun. Basically it's the it's the
39:03
usual nerd thing. We had a vision for with with my
39:07
partner about what agent frameworks should be like.
39:12
I actually started, I first started looking because I, I
39:15
assumed there was something there would be something there
39:18
and I looked at a bunch of them. Lung chain alumni didn't have
39:23
much back then Crew AIA, couple of the young, a couple of the
39:26
more more prototypey ones that were around back then.
39:31
You didn't find anyone that looked the work quite the way I
39:35
wanted it to. And in particular crew I came
39:39
closest. But then what?
39:42
But then they really wanted to have their own walled garden.
39:46
So when I when I submitted to them APR that would allow to
39:50
freely interpret with any kind of land chain or lama index
39:53
agents or any other agents, they they ignore the PR Oh, wow.
39:59
At that point I said, no, like what it's, it's a Lego set,
40:01
right? It's my favourite metaphor for
40:03
for this whole game. I want to mix and match.
40:06
And so that was a starting point.
40:08
And so now Motley Cruz core premise is you want to be able
40:12
to mix and match any frameworks at all.
40:14
From other Gen, Llama, index, LAN, Chain Korea, they all have
40:19
their strength and their weaknesses, so you should be
40:21
able to use the best tool for the job without trying to pull
40:23
people into a log garden. And then as we tried to use it
40:28
for things, which is the only way to make it good, we also
40:32
started adding other patterns that I haven't seen anywhere.
40:36
And my favourite one, for example, is a forest validation.
40:42
Oh, So what, what, what, what normally happens when you use an
40:46
agent with tools. For example, you have an agent
40:49
that generates Python code. Then if you want to make sure
40:55
that the that the Python code is that, then the agent calls a
40:58
tool. We stress, for example, to run
40:59
the code. And if there's any errors, it
41:01
gives them back to the agent. And the hope is you tell the
41:04
agent in the prompt that the agent will keep trying until the
41:08
agent will keep trying until the code is valid, until the tool
41:12
says you're good. However, there is no guarantee
41:16
of doing this and the LMS are famous for doing strange things
41:19
sometimes. So basically you're putting that
41:22
into the prompt and help. Whereas with force validation,
41:26
what you say is that the agent is only allowed to return result
41:31
via the tool. And so the agent tries to call
41:36
the tool with that say Python code.
41:38
If the call, if it's, if the code is fine, the tool returns
41:41
the code and if the code is not fine, it returns the reasons why
41:45
not to the agent. The agent tries again.
41:48
And if the agent tries to return directly to the user, the agent
41:51
gets told no, you have to return by calling the tool, try again.
41:55
And this way you have guarantees because then if you get anything
41:58
at all back from the agent, you know it's been validated and you
42:03
see this pattern, People reinventing this pattern with
42:06
like clammy index workflows and line graph and whatnot.
42:09
But I haven't really heard it described as a pattern
42:12
elsewhere, strangely enough. So you're, I like the force
42:18
validation description because it is very clear on how the
42:25
agent is not the one that is giving you it, it's the tool.
42:31
And by way of the tool, if you're getting it back, then you
42:35
know that it's it's been able to go through the tool and past
42:38
what it needs to go through. Exactly.
42:43
OK, so and the other idea on Mötley Crüe, which is really
42:47
cool is the and it also makes sense on the name you're using
42:52
any framework that you want. So how does that even look?
42:55
It's an abstraction above the lane graphs and the llama
42:58
indexes and autogens. So right now, first of all, all
43:04
of our well, we have wrappers for all the common agent types
43:08
because every framework has their own agent pairing class
43:11
and we can wrap them all, which is necessary, for example, to
43:15
make them all support the force validation pattern, which which
43:20
we can well search certainly LAMA index and long chain agents
43:25
we can make support. We support the force validation
43:27
pattern as well. Oh, they all support their
43:30
honourable interface. So you can plug them into land
43:32
graph as well, because land graph is actually cool.
43:37
And so so that doesn't those are the main 2 bits.
43:40
So you have wrappers and those wrappers also inherit from
43:43
landfills runnable because you can say many things about
43:46
Lancade which had not which, which had contained the word
43:50
flag planting and similar not entirely nice words, but to some
43:55
of the things it has is really cool.
43:56
And Landgraf is certainly one of them.
43:58
Yeah, yeah, excellent. Well, that's fun and that's
44:05
fully open source, so anybody can go and play with it right
44:07
now. Exactly, and the commitment is
44:10
to make it truly open source so it will never be used to try and
44:14
upsell people into stuff, or will never deliberately cripple
44:17
it to make people pay a little more for the pay.
44:20
Pay a little for the paid version.
44:22
It's meant to Max be Max, and that's also why it is maximum
44:25
compatible with everything. Like the next fun thing that
44:27
we'll have to look at now is Anthropics Model Context
44:31
Protocol. Oh yeah, it looks really cool.
44:33
So we really have to support that.
44:35
Very cool. Not there yet, but coming soon.
44:39
And now what's causal tune? First of all, can we just take a
44:43
a moment to note that you've been pretty busy creating a lot
44:48
of stuff over there. We've gone through what why is
44:51
pizza, We went through Mötley Crüe.
44:53
Now we're going to talk about causal tune and also just
44:57
everything in general that you're up to.
45:00
Hats off to you all at the Wise ML and AI section of that of
45:07
that company. Well, first of all, I've been
45:09
very fortunate to work it twice, which is very cool with people
45:16
doing other things in parallel outside of working hours.
45:20
And so now I've I've been able to officially lay down my people
45:25
leadership responsibilities and work part time and wise and
45:30
devote the rest of my time to bring up a start up and causal
45:34
tune is exactly the stuff I've been telling you about causal
45:38
inference segmentation. So the idea that in marketing
45:42
you cannot, you can extract more value from an AB test than just
45:46
averages. In fact, you can observe the
45:48
impact on every customer and use this for segmentation by impact,
45:52
targeting by impact and all those wonderful things which
45:55
until you've seen them done, you wouldn't even believe it.
45:59
So causal tune is the causal inference with marketing, but
46:03
it's also the segmenting piece because I know there was there
46:07
was a few different parts that we talked about within that.
46:10
Well, so causal Tune, it's an open source, it's a library open
46:13
source by Wise. So we are actually using it wise
46:16
by the way successfully with the numbers.
46:18
So we we do see a distinct upticks in click through rates
46:22
and such like. So causal 2 is just library for
46:26
causal difference. So it does 2 cool things.
46:29
Cool thing number one, it allows you estimate customer level
46:32
impacts. Cool thing #2 it allows you to
46:37
estimate outcomes of hypothetical treatments.
46:41
So suppose you did, you did some kind of assignment using your,
46:44
your causal tune targeting or just a random test even.
46:48
And then all of a sudden you're, you know, you go, you head of
46:51
marketing comes in and says, oh, no, you should have reused these
46:54
rules instead. And then instead of waiting for
46:58
another month to run another test to test those rules, it's
47:01
actually possible you're having, for example, a past randomised
47:04
trial result to compute with a very high degree of precision of
47:08
what the outcome of any rule would have been on that sample.
47:14
So if you want to test. So if you want to test ideas of
47:18
simple targeting rules, you don't have to run a new test for
47:21
each one. You just run a random test 1st
47:23
and then you use that data set and get the outcomes with
47:27
confidence intervals of any other assignment.
47:29
You could try it. And you don't actually need to
47:32
even be sending emails, you just run the tests on old data.
47:39
Well, you need to have one test. You need to have one test full
47:44
of randomised one will do perfectly and then using this
47:48
you can estimate the you can get the average outcome of any other
47:52
assignment or based on the same customer features that you had
47:56
in the original test without running more tests.
48:00
So why do people, for example, set aside randomised samples
48:04
whether they're targeting? It's just ways of sample size
48:06
when you can just do it exposed by math.
48:10
Yeah. OK.
48:12
And so now what you, you mentioned you kind of hinted at
48:15
it, you're building a start up with this.
48:19
What does that look like and. So very early stages.
48:23
We hope to have something up in a couple of weeks because it's
48:28
not just the product itself. So the technology itself has
48:30
been tested it Wise, it works, it's been open sourced by Wise
48:34
as well. So there is no obstacle to using
48:36
it in the start up by anybody, including myself.
48:40
And but now there's the whole machinery of of a SAS start up,
48:45
you know, hosting and to use authentication and all the best
48:49
that go into a functioning system.
48:52
Once we get those up, we really, we really would love to have
48:56
more people outside of wise try this out and see the benefits
48:59
because any kind of targeting where you know something about
49:02
the users who your target is, this is the technology with
49:08
which you can squeeze as much information as given.
49:11
We can do the targeting as well as it can be done giving the
49:14
features that you have. I actually think it's pretty.
49:17
It's pretty close to optimal. And how have you been using it?
49:21
Is it just more for if you're running a sale or if you're
49:25
running better rates or what are the, can you give me like an
49:30
example of what you actually do? Because I'm not sure I fully get
49:35
across the finish line on understanding like how that
49:38
looks with an e-mail that I get from Wise.
49:43
So in this particular case, the one that we have just got to
49:47
where the where the weather test is just over and we've seen nice
49:50
numbers come up. The wise doesn't just offer
49:56
transfers, it offers many, many nice things.
49:58
You have assets, you have balances, you have the card, and
50:02
you can use the card in different ways.
50:04
And so we have between half a dozen and dozen emails
50:08
encouraging users to use a particular aspect of our
50:12
offering, what we internally call the product.
50:17
And then the measure of success are people who actually go ahead
50:21
and not just click on the e-mail, but actually register
50:24
and start using the that particular aspect of our
50:26
offering within a certain time window of receiving the e-mail.
50:30
And then the question is, now that you have 8 or 10 different
50:36
encouragement for different festive part to choose from,
50:39
which one do you send to this user?
50:41
Yeah. And that's something which you
50:45
can try naive rules, but it'll where you'll get much better
50:50
impact if you do it with this kind of technology.
50:55
OK, OK. Yeah, that makes sense.
50:59
So it's like, hey, this person generally will use my use case
51:04
because I feel like I've probably seen these emails
51:06
before. And as I mentioned, I use wise
51:09
they're you're looking at me and you're saying I generally
51:15
transfer from American dollars to euros and I'm using the
51:21
savings function or the checking function right now.
51:25
And you know, what he might like is the credit card or the the
51:31
card, because I have thought about that.
51:33
And so I, I definitely have. And if an e-mail hit me in the
51:39
right moment, I probably would end up getting it.
51:44
Exactly. And that's that's exactly the
51:46
kind of thing. And the nice thing is that these
51:49
things are that now you can take into account many of the other
51:54
things like different regions might grow differently, or
51:57
people with different average transaction sizes might behave
52:00
differently. You can take any kind of
52:02
features into account. Just train your model and it
52:04
tells you know which is the most, which is the thing that's
52:07
most likely to produce a positive result.
52:09
Very cool. Well, can we talk a little bit
52:11
about organisational structures and what thoughts you have on
52:17
those? Oh, ABS, I would love to.
52:23
In fact, that's the big relation when I came to WISE was the
52:28
degree of autonomy that not just a person but a team could have.
52:32
And when I first joined the Wise Treasury, it was really a
52:36
revelation where to be in an organisation where certainly all
52:41
I saw around me is that nobody told anybody else what to do,
52:45
not even your lead, which is strange when you hear about it,
52:50
but it actually works. And so this idea of autonomous
52:55
organisations is very close to my heart since because it's not
53:01
just about how people behave, it's just about intent.
53:03
It's also about how organisations are structured.
53:05
For example, if you have a vertical IT organisation which
53:09
everybody has to compete for, you can forget about autonomy.
53:13
And if you have or if you have an organisation, if you have a
53:18
team which is only regarded as a cost centre, then this team will
53:21
penny pinch to the detriment of the rest of our organisation
53:24
because that's her incentives. And so there is a personal and a
53:28
structural component to it. And I've actually been working
53:31
on a book. Let's see how well I get about
53:35
this. So reasonably far advanced,
53:39
hoping to bring it out early next year, but we'll see.
53:42
Nice. And the fun part about the
53:45
timeliness of this is that I think, and I'm not the only one
53:50
thinking this, of course, that the, and yet in this way of AI
53:57
will transform the way organisations are structured,
54:01
right? If you think about it, a typical
54:03
firm, sort of like a big firm, is structured around is like
54:06
information flows and is information flows a hierarchical
54:10
because that's until recently, that's the only way that humans
54:13
knew how to deal with text data. You have middle managers making
54:17
reports for their managers, who are making reports for their
54:20
managers. And then many Chinese whispers
54:22
later, the CEO thinks they know what's going on.
54:25
Yeah, but now if you can shortcut this whole, this
54:30
pyramid of Chinese whispers and just have AI look directly at
54:36
all the raw data in the organisation.
54:39
And if necessary go out and ask questions over chat when it
54:42
doesn't know enough and then give you the answer, the right
54:46
granularity, maybe you don't need the hierarchy.
54:50
And so that's, that's a very, that's a very, very exciting
54:54
thought. And there's certain space that I
54:56
will try to be present in over the next over the next years.
55:00
You did say something that was a question that came up time and
55:04
time again when we did the AI Agents in production conference
55:07
a few weeks back, which is how do you create an agent that
55:14
understands when it does not have enough information?
55:18
And that's a really hard problem.
55:20
I think like you just said, because AI and you, you
55:24
mentioned it before, like if you give AI and MLA task to find
55:29
something, it's always going to find something, whether or not
55:31
it actually is the thing that you're looking for, it's always
55:35
going to find something. So it's really hard to have an
55:40
agent understand that it does not have sufficient information
55:44
to answer whatever the question is or whatever the report it's
55:48
drawing up. It will.
55:49
A lot of times that's when it defaults to hallucinating.
55:55
That is true, but at the same time I think you know there are
55:59
solutions for this. You have to build.
56:02
You have to build to this from the start.
56:04
I mean, the simplest 1 is consensus.
56:08
If you run it several times, does it come up with the same
56:11
answer or different answers? And if it does come up with
56:14
different answers, then maybe it doesn't know.
56:17
And I'm 100%. But frankly, as far as this kind
56:22
of technology goes, I'm happy to be a fast follower.
56:26
I'm 100% sure that smart and well paid teams at Google, Open
56:32
the Eye, etcetera are even now working on smarter decoding
56:36
methods to deal with this because it's such an obvious
56:41
blind spot in LMS and I'd rather wake another quarter to quarters
56:45
and use that to build products. Yeah.
56:49
And it was a bit of a tangent to the actual organisational
56:53
structures conversation that I wanted to have, but you said it
56:56
and my mind went that way like a little dog chasing a ball.
57:01
The organisational structures piece though for me is
57:05
interesting because AI can imagine a lot of folks that are
57:10
listening, including myself. They think, well, how does
57:13
anything get done and how do real initiatives?
57:18
Who's crafting the high level vision or the goals if nobody is
57:23
telling anybody else what to do? Well, I think there is still
57:30
hierarchy, it's just that the hierarchy is not coercive and
57:35
it's just that the the point of a lead is not telling people
57:39
what to do. It's really instead it's the
57:42
following. Firstly, it's telling people a
57:45
story. Why is the team here?
57:47
What is there for in big terms? How does it fit in with the rest
57:50
of the organisation? That's task number one.
57:54
And task #2 is this is this is really enough for smart people
57:58
to figure out what is most important for them to look at.
58:02
And the other half is then clearing out of the way
58:05
obstacles the team can deal with itself.
58:07
So if there is an organisational problem somewhere, somewhere
58:10
else in the organisation, some stupid process is stuck or you
58:14
know, things happen. But then just going in there
58:17
cleaning this up, I think those are the, the big, the two
58:19
biggest things. And if a leader each level does
58:22
this, and especially the storytelling part, then the
58:28
amount of freedom that is unleashed thereby, the amount of
58:31
creativity is astounding and it even has an economic effect.
58:35
So when I was hiring for Wise Data science team, it was not
58:40
uncommon to people turn down higher offers elsewhere because
58:44
they came here for the autonomy. And the autonomy in practise
58:51
means that I, if I was on your team, I can say, you know what
58:58
this project, considering what our story is and why we're here
59:03
on this team, I feel like the best way to help us move the
59:07
needle on this project or on this team is to do this project
59:13
and I propose something and then it gets sign off.
59:16
Or is it just I run with it and I say hey I've already hacked
59:21
together a little bit and now I need more resources from XYZ
59:25
teams? I guess, and that's where I
59:28
guess that's how seniority works in autonomous teams, because
59:35
seniority is measured by the amount of people that you're
59:39
able to bring along with a for your with your idea.
59:44
So if you're really a junior and you don't really know what's
59:47
important, then you're you're generally happy to take guidance
59:50
and that is you progress you first.
59:52
So you have to convince people around you that what you're
59:55
proposing makes sense, at the very least makes enough sense
59:59
for you to work on it. But then how you grow to be more
1:00:03
senior is when you start saying you start convincing people that
1:00:06
it's an important nothing for the whole team to work on it.
1:00:10
And the more the more people you can, the more people you can
1:00:14
engage and bring along with your storytelling, that the more the
1:00:17
more senior you are. And then eventually titles
1:00:21
adjust. And what about this scenario?
1:00:24
I am very excited about working on a project like, let's just
1:00:31
say Wise Pizza. Before Wise Pizza was a thing.
1:00:35
I go around trying to rally the troops and it falls flat.
1:00:41
People don't understand it. People aren't really interested
1:00:45
in joining the cause. How long do I have to try and
1:00:52
champion for that before it falls flat completely and I give
1:00:56
up and I'm on to something new? Is it a matter of days, weeks,
1:01:01
months? Or is it just something that I
1:01:04
put on the back burner and I never truly let the dream die?
1:01:09
Well, first of all, of course everything I say, Needless to
1:01:12
say, is my are my opinions about how an autonomous or
1:01:16
organisation offence around autonomy would would really
1:01:18
work. And in any particular
1:01:21
organisation, including wise many people would disagree
1:01:23
that's how organisations and humans work.
1:01:26
But in so I would say that since there is no objective standard
1:01:32
of truth of usefulness for most people, like maybe marketing
1:01:37
people, they increase revenue, but most people in our
1:01:40
organisation have no direct measurable impact on their own.
1:01:43
Then you have to bring you have to deliver enough value that
1:01:50
makes sense to people around you.
1:01:54
And but you'll, you'll have to have some slack to pursue things
1:01:58
which are to pursue things which make might make no sense to
1:02:01
them. But you have to have the more
1:02:04
record you have of delivering things that visibly add value in
1:02:08
a way that makes sense to others, the more slack you will,
1:02:10
the more slack you will gradually get for doing things
1:02:13
that might have make sense at once.
1:02:17
But it's all, it's very much your relationship.
1:02:19
It's a human thing. That's I guess that's the thing
1:02:21
about autonomy, whether like it's not, you can think about
1:02:25
like my favourite metaphor is that of a machine versus a
1:02:30
forest, right? So many advice, so much advice
1:02:33
on scaling and organisation centres around making it like a
1:02:36
machine. Every role is precisely
1:02:38
described. People are replaceable and then
1:02:40
it can scale. Whereas autonomy centric
1:02:44
organisations are more like gardens where things grow the
1:02:49
way they are and they're just around each other.
1:02:52
But it's really, oh, there is no No2, no two plants are alike,
1:02:56
No2 humans are alike. And then you have to call you,
1:03:01
you have to tell stories to the people around you and you and
1:03:05
what you have, what you're doing must make sense to them.
1:03:09
And that's and that's the main criteria.
1:03:12
Are you familiar with permaculture and that whole
1:03:15
movement for gardening? Which one?
1:03:19
Permaculture. I am not oh what is it about?
1:03:23
Talks about how and, and I'm by no means an expert, but from
1:03:27
what I understand is is not only if you look at traditional
1:03:33
farming and traditional gardening, you'll place all the
1:03:37
tomatoes in a line and or maybe you'll have a whole field of
1:03:42
corn, but this is more like, you know what goes really well with
1:03:46
tomatoes is basil because it keeps the fruit flies away.
1:03:52
And so you put tomato, one tomato plant and then one basil
1:03:57
plant and one tomato plant. And so you're varying things and
1:04:00
you put different plants together because they have a
1:04:04
nice ecosystem or homeostasis together.
1:04:07
And it feels like that is what you're talking about.
1:04:12
It's, it's not only gardening and having each individual plant
1:04:16
be an individual that grows in the way that it does.
1:04:19
But if you can put two plants together that work very nicely
1:04:24
together, you're going to get that combined effect and the
1:04:28
outcome of both plants doing better.
1:04:32
Oh this is a thank you for a lovely metaphor.
1:04:34
In fact this is perfect because also in terms of autonomy,
1:04:39
functional verticals are the kiss of death.
1:04:43
So a centralised IT reporting line which overrules product
1:04:47
centric reporting lines is the gist of death for any kind of
1:04:50
autonomy. It's exactly this.
1:04:52
You have to make the different specialties and proportions that
1:04:54
allow them to do the thing that they had to do and without
1:04:59
having without having any kind of vertical prior vertical
1:05:04
priorities getting in the way of.
1:05:06
Brilliant.