Latest / Agentic Conversations (formally mlops.community) / Enterprise AI Adoption Challenges

Transcript

Enterprise AI Adoption Challenges · Agentic Conversations (formally mlops.community)

0:00
So let me start with the metrics on the productivity side.
0:03
You want to make sure that you keep your power users.
0:06
These are the guys who are going to really push you to the next
0:09
level. Users right now scream of oh, we
0:11
want Google Drive. What does that mean?
0:13
They're like, I don't know, but I want it.
0:15
We're now in the loop of can we get this to work at all?
0:18
Can we make it something that users can easily set up?
0:22
And then can we make it so intuitive that it's going to be
0:24
a no brainer? We talk with Sean and Paul, both
0:31
employees at Process. Paul is the VP of AI and Sean is
0:35
the product lead for Tokan. I myself, I'm Dimitrios, the
0:39
host of the ML OPS Community podcast that you're listening to
0:42
right now. Let's get into this
0:43
conversation. This episode I wanted to view
0:48
from a product lens and Tokan has been built and it's had a
0:53
lot of iterations. It's also had a lot of learnings
0:56
specifically around how to get users to use AI products.
1:00
Yes. So, you know, just to kind of
1:04
sketch the environment, you know, the group that we work at,
1:06
pros is right. So it's a big global group with,
1:09
you know, about 100 companies part of it, some very large like
1:12
OLX and Swiggy and I, food and Delivery Hero and others, and
1:16
some smaller. But you know, at the center here
1:19
at the AI team that we have built here in Amsterdam, our
1:22
mission is to help the entire group adopt AI everywhere.
1:25
It makes sense. And so when in the early days,
1:29
these large language models, you know, we were partnering with,
1:31
you know, many of the big labs out there, you know, before
1:35
ChatGPT and so on to kind of understand where this was going.
1:38
And it was very evident, at least to us that this, this was
1:41
sort of this technology was on a acceleration path.
1:44
And so in the early days, we basically started to make these
1:47
language models available through Slack for people to kind
1:50
of experiment with them. And eventually that grew out to
1:52
be Token. And that's now today it's a
1:55
basically a productivity tool and even starting to become sort
1:59
of a platform for folks to build on top of all the language
2:03
models that we continuously expose through Token, not just
2:06
in Slack, but also, you know, in a web interface if people want
2:09
to chat to it and so on. And now also through AP is to
2:14
build on top of all of these large, large language models,
2:17
generative models for images and other agentic systems that are
2:21
sort of powering Token and many of the use cases across the
2:25
group. What are some product metrics
2:27
you've put around the usage of token?
2:29
Yeah, they're, they're basically 2 buckets.
2:31
So the first is around productivity and adoption of
2:35
Gen. AI across the workforce.
2:37
So we believe that it's really important for everybody to, you
2:41
know, play with actually for them to understand how these
2:45
models work, what they can and can't do.
2:46
And the only way to do that is we found, is for people to try
2:48
it, right? Because the technology changes
2:50
so fast, you can't just design a course and put it on, you know,
2:54
our internal learning portal and say, learn prompting.
2:56
It's going to make it done right, So these models you
2:58
should. Use because that course has a,
3:00
you know, a half life of three weeks yeah.
3:02
So that's the first bucket. So there's a bunch of metrics
3:05
which I can highlight there and the second is around adoption
3:09
for actual consumer facing or internal use cases across the
3:13
group. So we're very large group, 2
3:14
billion users that we serve in different parts of the world.
3:18
There's a ton, you know, we've got thousands of ML models and
3:21
use cases that could be powered by some of the same systems that
3:25
we're building for token that are agentic, that use the
3:28
latest, the latest large language models, the the tool
3:30
calling and so on. So let me start with the metrics
3:34
on the productivity side there. We primarily focused on number
3:39
of users that tried token. So how many people have actually
3:44
tried token? We'll be trying to get to
3:46
basically more than 80% per company of people that have
3:49
tried it, because there's a big hurdle.
3:50
Just try asking O can a question related to your work.
3:55
You know, how do I summarize this?
3:56
How do I translate that so people start to do that?
3:59
The second is actual frequency of usage.
4:04
An so there's a really interesting, let's say evolution
4:09
we've seen happening is, is how people go from asking it, you
4:12
know, casually once a week, once a month, a question to becoming
4:16
super users. And I can share some stats
4:19
around making sure that, you know, we see growth of super
4:21
users who is that's basically the category that asks is more
4:24
than five questions a day. And and then the third is total
4:30
number of questions asked across the group.
4:32
So those are the kind of three metrics we focused on for, for a
4:35
long time. And then on the on the second
4:36
bucket, which is more on the adoption of use cases for not
4:41
productivity, but everything else, right.
4:43
So where you put things as part, we put the token engine as part
4:47
of your workflows through an API.
4:49
And then we measured API calls and also a number of individual
4:54
use cases that we're seeing, you know, using the technology
4:58
underneath to kind of engine as we call it.
5:02
It seems like this five questions a day is like your
5:06
North Star metric, because if you can get someone above that
5:09
then they are now a power user and you want as many people to
5:12
become a power user as possible. Is it like that?
5:14
Like it reminds me of I heard a stat back in the day with
5:18
Facebook, they would try and get someone who's on boarded to
5:22
Facebook, at least seven friends and once they had seven friends
5:26
they were hooked. Yeah, I I definitely think that
5:31
that there is that sort of turning point where now you've
5:34
become a regular user of these tools, right.
5:37
And by the way, just to be clear, like we encourage people
5:40
to use all the tools. So kind of the in house one
5:43
which uses, we know we swap in and out models all the time so
5:47
people can experiment with them and we learn how which which
5:50
ones works better and and not. But we also encourage them to
5:53
use all the other tools, right. In fact, if you look at my
5:55
browser, I've got, you know, basically one, one browser with
5:59
all my AI assistance open, right?
6:00
That includes Claude, it includes ChatGPT, it includes
6:03
grok with AK includes, of course, token.
6:06
It includes Gemini. And now I've got Devin and a few
6:10
others that I'm testing with. And you do see and and so the
6:14
five questions a day for token, it wasn't as scientific as maybe
6:19
the seven friends on Facebook, but you know, once you become a
6:22
daily user. So your first intuition is, hey,
6:25
should I ask this question? The token, that is the kind of
6:29
thing you want to achieve. And then it doesn't matter if
6:31
it's Tokan or Cursor or we want people to become a sort of
6:36
natural users of these tools and help them do their work better,
6:39
faster and more independently. How are you measuring that?
6:45
Because I'm sure there's a lot of stuff that people are using
6:48
that are outside of Tokan, but they're still using it and you
6:52
can't really track it. Or can you?
6:53
Yeah, we can. I mean, that's for all teams.
6:55
So obviously for, for our team, we, we, we, we track all sorts
6:59
of things like how many PRS are being, you know, generated with
7:02
the help of AI, how many PRS are accepted or commented or involve
7:09
some kind of AI, Right. And the, and the 1st days it
7:12
was, well, you know, how many people are actually using a
7:16
cursor or are using a certain set of tools?
7:18
How many copilot users do we have actively?
7:20
So there are things we in my team that our team that we can
7:23
definitely track and, and do track across the group.
7:26
We don't have that visibility everywhere, but we do.
7:30
We've done sort of experiments where we give part of a team
7:33
access to one of these tools. That's in the early days of
7:35
copilot, we had done that and then the other team not.
7:38
And then we do surveys and see how often they do they use it,
7:41
what's the difference? What is the feedback?
7:44
And just for us to understand, because, you know, there's a lot
7:47
of independent research that also gets done.
7:50
It says, you know, engineers have an average of a
7:53
productivity increase of 20%, but that once you peel that the
7:58
layers a little bit further is like which types of engineers
8:00
for what types of tasks is at 20% overall?
8:04
Or is it just on those tasks, you know, used it's, it's, it's
8:07
much harder to really pin down what the value is.
8:10
And we're pretty convinced that the value is real and that it's
8:13
in the double digits. We care less about whether it's
8:17
21 or 23 or 41 or 42 or for whatever.
8:21
We just want to make sure we, we, we make it easy for people
8:23
to adopt if they want to and then also have a good
8:27
understanding of where we want to encourage people to adopt
8:31
certain tools. Which types of, you know, we've
8:32
got big engineering teams, right?
8:34
Thousands of people across the group that are building
8:36
software. We've got big marketing and
8:38
sales teams. We've got, you know, basically
8:40
pockets, customer support and so on that where it's natural for
8:44
us to kind of make sure that those people, they can do their
8:47
work better, faster and more independently.
8:49
And we want to understand and that's why we measure these
8:52
things. And once we see that the value
8:54
is real, we start to sort of encourage and push for adoption
8:57
more actively. Is that what you talk about when
9:00
you say like the AI workforce? Yeah.
9:03
So AI workforce indeed has as a big productivity element to it.
9:10
So how do we make sure people become more productive?
9:13
But like I said, it's really hard to, to kind of quantify
9:16
this second value of people using AI.
9:19
But it's so important in a tech company like ours is they need
9:22
to have that intuition, right? So if you, you know, if, if the,
9:26
if, if the lawyers and the folks in finance and HR and marketing,
9:29
all the different functions don't understand AI well, it's
9:32
much harder to do all those other, you know, forward
9:35
thinking products and features and, and, and, and actually
9:39
trying to think about how do we want to build, you know, a
9:43
better delivery experience, a better online shopping
9:46
experience, an online travel booking experience.
9:49
So the intuition having that is it's almost like a culture
9:52
change, right? You need to have that on top of
9:56
the productivity benefits which are, you know, an obvious value
9:59
as well. All right.
10:01
So let's bring on Sean now to talk about how he's been
10:03
encouraging adoption for Tokan because he's been the product
10:06
lead and I know he thinks a lot about these trade-offs and what
10:11
to build for the power users versus what to build for the
10:14
general public or the general employee.
10:19
I'm Sean. I'm the product lead at Tokan at
10:22
Process, part of the Pros AI team, and I take my coffee and
10:26
overdose and I, we actually got a barista in the office and I
10:30
started to only take machine coffee because I find that it
10:34
takes the joy and the magic out of the weekend coffee.
10:37
I like it. What were you saying about the
10:40
stigma? I think when Jenny I started or
10:43
when it started coming out and you used it in order to complete
10:47
a task, you're always trying to hide it because it always felt
10:50
like the cheap way out. That's so true.
10:52
And I think that's still there and, and anywhere, right?
10:54
I mean, it's, oh, this is a Devon pull request.
10:57
It's not my pull request or this is a this is something that I've
11:01
written with token and you always feel like this looks like
11:06
AI generated, which can be perfectly fine content wise, but
11:08
there's there's a little bit of not not even from the other
11:11
side, but at least in the back of my mind and many tasks like
11:13
and then you're like, no, no, this is what we built it for.
11:16
It should do this but there. It's one of those things that I
11:19
contemplate a bunch on the Doorman Paradox, as I've heard
11:23
it called, where a hotel got rid of all its doorman and then it
11:28
thought, wow, we're saving money, this is great.
11:31
But later they found out that people started loitering in the
11:34
front. Also, the guests weren't having
11:37
the best experience because the first thing that they were hit
11:39
with when they entered the building was they were lost.
11:42
And they realized that a doorman does so many different things
11:46
than just opening a door for someone.
11:48
You mean that in terms of customer touch points or in
11:50
general? Well, I, I think about it with
11:52
AI where we are so quick to try and use it, but we don't think
11:58
about the secondary and 3rd order effects of when we're
12:01
using it. And one of them is this.
12:04
In my mind, I, I draw a parallel to, oh, people look at it and
12:09
they think this is AI generated. And maybe you lose trust with
12:13
someone or it's not the same kind of outcome that you would
12:18
want, especially when you see so many people toting, oh, man, I
12:25
got my content marketing flow perfect because now AI does it
12:31
all. And it's this incredible
12:32
pipeline, blah, blah, blah. But then you look at the content
12:35
that comes out and you go, this is absolutely shit.
12:38
You're just ruining your brand by putting this out there.
12:41
It might be great that you can automate it all, but nobody's
12:44
reading what you're actually creating.
12:46
Yeah. And I think it's, it's very much
12:48
a question of when and intent wise how you want to use it.
12:55
I think deep research is an is an interesting example.
12:58
If research is one of the steps in the job that you're trying to
13:02
complete and you use deep research for that, it might get
13:05
much harder to complete the steps down the line because you
13:08
don't actually know all the considerations that have gone
13:10
in. And I think for many tasks,
13:13
it's, or many users actually, it's how you use it.
13:16
And many users generate something without reading it and
13:18
then do something with. And it's like, that's not how
13:20
you're supposed to use it. It's not there to just generate
13:22
blindly something. And so one of the things that
13:25
we've been trying to teach users is to really use it like a
13:29
buddy, like, hey, if I were to go do something, how would you
13:32
advise me go about that, right. And so more on a discussion
13:35
level. Of course you can have the
13:36
execution afterwards, but you should be very confident that at
13:38
least it's going to the right direction and then you can look
13:42
at OK quality wise. Of course you still need to read
13:44
it. You need to make sure that what
13:45
is there is actually useful. Let's talk about the evolution
13:48
of Tokan, because now you're starting to see these clearly
13:53
trotted paths of how people are using it.
13:55
Like you just said, use it as a buddy or use it for data
13:59
analysts, use it for X. It wasn't always like that, and
14:04
I think you all started working on it two years ago.
14:07
It has been a long journey. What can you tell me about,
14:11
like, just summarize the evolution?
14:13
Yeah, yeah. So let's start with probably the
14:16
question of why our app exists in the 1st place.
14:19
And this I wasn't that part of the team at the time, but the
14:23
team created a Slack app before I think GPT 3 came out because
14:29
the team started seeing these models come out and be like,
14:32
well, they look really interesting, but it's still
14:34
quite hard to find moments where we can use these in business.
14:38
What can we do? Well, the easiest thing might be
14:40
to just get create an app where anybody can interact with them
14:43
and we'll see what people do. And from there we then figure
14:46
out what we do next. And so we started creating a
14:48
Slack app at the time, this was again, far from agent to get
14:54
like there's nothing like that was around.
14:55
So we had some basic intent detection that routed I think
14:59
four or five different kinds of work.
15:01
And then that was spread through the portfolio companies that we
15:04
have at least some the ones who wanted to adopt it.
15:06
It was very organic, but it grew a lot faster than we thought it
15:09
would. And at that point we decided,
15:12
well, OK, let's actually turn this into a product.
15:14
And we spent a lot of time on critical tools that that are
15:19
needed in order to complete tasks.
15:21
Like they're now pretty commonplace, but the ability to
15:23
work with any kind of file, generate files and data analysis
15:27
came. So, so these things we worked on
15:29
quite early still on the intent detection model.
15:34
And then we just kind of tried to bring that more into, OK,
15:36
well, how do we take this experimental app and make it
15:39
something that people can actually complete work with?
15:42
And that we then converted into an agent that that seemed to
15:45
quite drastically outperform even at the time, the intent
15:48
detection. And then from there, we've just,
15:50
what do you mean by the system? Changed from intent detection to
15:54
an agent. So we had a system where when a
15:56
user asked a question, we would route the request down a almost
16:00
like a workflow. And so we would have to figure
16:02
out what intent the user had and we were quite limited in them.
16:06
Out of the tents we could add and we would if we wanted to add
16:10
a new intent, do some new fine tuning and training of a model
16:13
to do the routing. And so of course with an agent
16:15
you just add quote UN quote just, but you add tools and it's
16:18
it's easier to manage and expand over time.
16:21
Up until I think like 6 months ago, you all had you, you all
16:27
just started doing a ton of updates.
16:31
Can you talk to me about the last six months and what you've
16:33
been doing? So right now I'm the product
16:35
lead for Tokan, but when I started I actually joined kind
16:39
of as an adoption advisor. So my role was to help the
16:44
companies, help teams and companies and the employees
16:46
there to actually use the product.
16:48
So we we found out quite early that there is a set of companies
16:52
in the portfolio who were adopting this tooling a lot
16:56
faster than others. So we knew that there was
16:58
potential and room there. Any idea why that is?
17:00
Yeah, yeah. So this comes down to to quite a
17:03
while or not that much, but the nuances are difficult to figure
17:06
out. But much of it was almost top
17:08
down culture, like it's a, OK, let's give this a try, let's see
17:11
where it goes. It's very much a mindset
17:13
difference of being stubborn enough to make it work, wanting
17:17
it to work and being excited by all the opportunities that come
17:19
from it. And so it's very much, at least
17:22
on a larger scale, adoption level, it's very much a cultural
17:24
thing of how does your company view this kind of technology?
17:28
And a lot of it is driven by leadership and the way it's
17:32
encouraged because it's very easy to say, Oh yeah, AI is
17:35
important. But then there's no, there's no
17:37
policy, there's no guidelines, there's no instructions around
17:39
that. And that makes users not feel
17:41
very encouraged. They're just more uncertain.
17:44
And so building certainty around, hey, we don't just want
17:49
you, but we expect you to spend time on these tools.
17:51
And in the beginning, it's going to cost time and that's fine.
17:54
But it's your responsibility to learn.
17:55
And here's the resources that you have in in order to learn
17:58
it. But at the end, it's on you in
17:59
order to figure this out. And you should be interested in
18:02
this, not just because it's good for us as a company, but also
18:04
because it's good for you. It's a skill that will only
18:06
become more and more important. And so the sooner we start, the
18:09
better this is going to get. The cultural aspect, I like
18:12
that. And there's a, there's a certain
18:14
set of flexibility and, and nimbleness in there because, and
18:19
it also is part of the culture, but it's not just the adoption
18:22
and go and try it. But this inevitably changes how
18:26
you work. And that's something everybody
18:28
says, but, but in reality, often times, especially these kind of
18:31
new technologies conflict quite heavily with existing processes.
18:34
You do something in a certain way and it's not always easy to
18:37
just plug in, plug out, but you might have to rethink everything
18:41
that you're doing, which has some teams and some people are
18:44
more comfortable with than others.
18:45
And generally speaking, the ones who are more willing to
18:48
experiment to just see, hey, does this actually work?
18:50
Does it help? And then are quick to say yes or
18:53
no. They're the ones who benefit
18:54
from this a lot. And then the secondary effect is
18:57
if you have a subset of those teams available, then you then
19:02
you want to look at sharing. And so that's the second part
19:05
where it's quite painful for individuals to do a lot of this
19:08
discovery because it is lots of it is still quite technical and
19:12
it's very difficult and most of it isn't particularly user
19:14
friendly. Actually, agents are kind of the
19:16
opposite of user friendly and at least in their earliest
19:19
iterations. And so building a community of
19:22
sharing is, is super important because that means that if one
19:26
user discovers something now, the next 10 don't have to make
19:30
the same mistakes. And the difficulty, it goes back
19:34
to stigma is asking questions, asking for help on things like
19:38
this where you feel like everybody is 10 miles ahead
19:40
feels so stupid. But oftentimes everybody
19:42
struggling with exactly the same thing, right?
19:44
And so, yeah, you see these posts.
19:46
Oh, I've done all my marketing, et cetera.
19:47
But how does it really work, right?
19:49
How can I do it for my work and to having a kind of an open
19:53
culture to share and learn together?
19:55
I've heard people tell me that that are in this field.
19:58
They say I don't know if I should be learning RAG or agents
20:02
or tool calling. I feel so behind and in my
20:05
day-to-day. I'm working on a recommender
20:07
system but I am not sure if I can plug in AI in some way and
20:14
so that FOMO is really. Real.
20:19
The the culture is very much an enabler and an accelerant, but I
20:23
think, and that's a lot of what I spent my most of my time,
20:27
especially in the beginning on this is the actual education
20:30
part, because it's very easy to tell people, Oh, go and try and
20:32
use it, but it's very, very hard for most people to do that
20:36
right. It's not not it's a non obvious
20:39
you don't even know how to start you at some point I gave a talk
20:43
on how to build product using or how to build AI products for
20:47
users that actually work and are useful.
20:49
And one of the things that come up so often, and many companies
20:52
do this and part of it is early experimentation.
20:54
But you get a chat bot and there's a chat chat input field
20:58
and it says, this is chat pod X ask it anything.
21:02
And as a user, you're like anything, right?
21:04
It's like, so you ask the first thing that comes to mind.
21:07
And usually it's not something that's helpful to you.
21:09
It's just the first thing that comes to mind.
21:11
And even if it's helpful to you, there's no guarantee that the
21:14
agent or whatever chat pod X is able to do it.
21:17
But you've no, no, if no other way of figuring out other than
21:19
throwing things at it. And then it's not going to like
21:23
you'll it'll fail and you'll be like, well, ask anything.
21:25
Clearly that's not the case. So I'm just going to ask nothing
21:27
because doesn't work. And then you lose a user and
21:30
that user experience is not ideal.
21:32
I've been banging this drum of like chat is not the ideal way
21:37
to interact with AI and I wish there were easier ways.
21:41
I know that you have thought about building products like you
21:45
said you gave a talk on it. You also saw some user adoption
21:50
challenges, and so on one hand, we've talked at length on how
21:56
can you build more support for people who are trying this so
22:01
that they can have that stubbornness to breakthrough and
22:04
realize the value of it. Can you talk to me a bit about
22:08
some of the things that you did to try and help folks get
22:13
through that filter? Yeah.
22:15
So I think in terms of interface, we've done a few
22:21
things like follow up questions. I think lots of it is kind of
22:23
experiments around chat interactions.
22:27
The fact of the matter is that the chat is the kind of
22:29
predominant also in the way people think about it.
22:32
And so changing it now again, it's going to make it even
22:34
harder for many people to understand it because many are
22:37
just getting to the chat part right.
22:39
And so now if we change again, we'll, we'll break something
22:41
else. I think for us it's been a
22:44
pretty tight balance between we want to move fast with the
22:47
product. Everything is happening all the
22:48
time. We have a small team, so we need
22:50
to be able to build new capabilities, enable our power
22:54
users. And so we've actually spent a
22:56
lot of time in hands on support. I think in the beginning I did a
23:00
live webinar every two weeks with users and we did lots of
23:05
hackathons and workshops and training sessions.
23:08
We have the benefit that in reality we have a pretty
23:11
contained small audience, right? I think we we look at about this
23:13
30,000 ish employees and so I'm not.
23:17
Not so small. No, no, but I'm not too worried
23:19
of like, oh, we will have a potential of, I don't know, 20
23:22
million people out there that we need to educate, but it's, it's
23:25
something that's much more feasible to do.
23:26
So we organize big events like we had a token day and we
23:30
organize training sessions, we organize cross company team
23:33
events. So a lot of it has been kind of
23:35
hands on because it's a very easy thing to keep talking about
23:39
and there's parts that we can build in the platform.
23:41
But what I, what I really wanted to get over is the initial
23:45
reluctance of fear of using something.
23:47
And some of these, some of these webinars were just, I'll give
23:51
you one that that actually turned out to be really funny or
23:54
funny in the sense that it worked because it actually
23:56
started as an accident. We had a case prepared and we we
24:01
joined with a company that just got access to Tokan or I thought
24:04
they did. And so we we joined, we joined
24:07
this webinar. I think there were about 200
24:08
people and it was a hands on case, right?
24:11
So I'm like, OK, here's the case.
24:13
This how it works. We introduced a bit of the
24:14
product before, but like just go try you have an hour.
24:17
Our hypothesis was nobody takes time to experiment and try.
24:21
So if we get them to try in a dedicated safe environment, then
24:24
we get the first kind of fear response out of the way.
24:28
And now people are much more comfortable to start.
24:30
OK. But in that session
24:31
particularly, it turned out that some part of the onboarding had
24:34
gone wrong and so they didn't have access.
24:36
And what I then decided to do in a kind of in a mild moment of
24:40
panic of having 200 people sitting in a webinar being like,
24:43
OK, what are we going to do? I just did the case live myself
24:47
and it was both one of the best received webinars as well as one
24:51
of the highest user uptakes afterwards because the way I did
24:55
it was just kind of walk through, OK, now I'm thinking
24:57
this is how I'm going to use it. And this is.
24:59
And so just this kind of life interaction, almost like an
25:01
example, even if the case isn't super relevant for the user
25:04
turned out to be incredibly appealing.
25:07
And so just going back to the product question, we've we've
25:10
done some experimentation on the product and part of why we've
25:13
enabled our app and the web as well or we've done a big push
25:15
for the web is to make it more use usable and user friendly as
25:19
well. It's like it can be a bit
25:20
tricky, but a lot of the education has a lot of the early
25:24
user guidance has come from actual training and guidance and
25:28
materials. It's almost like you have one
25:32
aspect, which is you're trying to make the technology and the
25:37
user interface as intuitive as possible, but then on the other
25:41
side, you're evangelizing as hard as possible to have folks
25:47
jump on the train and get involved.
25:49
And I know there's this trade off between building for the
25:54
power users versus building for everyone else and making it as
25:59
intuitive as possible in the lowest common denominator versus
26:03
adding all the features that the people that are ahead of the
26:06
pack want. How have you looked at that?
26:10
Because it seems like it's going to be hard to have wide adoption
26:16
if you're constantly chasing and adding features for the people
26:19
at the front of the pack. Yeah, Yeah, it's a, it's a
26:21
difficult balance because you want to make sure that you keep
26:27
your power users, they are your most important users in, in, I
26:31
mean in a, in a company political way in terms of
26:34
metrics and impact. But it more in a product way
26:37
because these are the guys who are going to really push you to
26:40
the next level. And they're the ones who will
26:43
figure out what could work, how that could look.
26:46
And they're the ones who really give you good feedback and they
26:49
will educate all the other users if they're able and willing to
26:52
share. And so you really want to keep
26:55
those users around. And if you don't build the next
26:58
thing for them, you risk them going away.
27:01
So that's that's kind of always been forefront of mind.
27:04
And then it's been what's the minimum product guidelines and,
27:09
and usability that we need in order to get out of users on
27:11
board. And we've, we've under an
27:13
overshot. And so we course correct as we
27:15
get user feedback. But I think the balance has been
27:18
mostly what's the minimum to get users started while we build the
27:22
next big thing. And the the cycle for building
27:25
the next big thing usually is, can we make it work at all,
27:29
right? Is that something that we can
27:30
make work because, and now it's somewhat commonplace, but at
27:33
some point in the past, we build a text to sequel agent as a
27:37
verticalization on the platform. And the first question wasn't,
27:40
oh, how do we make this useful? How do we make it intuitive?
27:42
It's still neither of those that the question at that point in
27:45
time was, is this feasible at all?
27:47
Right, Can we make this work? OK, then we made it work, but we
27:51
did it by ourselves building it. So now we're like, well, we are
27:54
not data experts, the power users, the data engineers, the
27:57
analysts or the end user and commercial that once data,
28:00
they're the experts in their domain.
28:02
So now how do we build this platform in a way that they can
28:05
use it instead of us setting up everything?
28:07
So then that's the next step. And now at some point down the
28:10
line you would look at, well, OK, how do we make this super
28:13
easy to use and super intuitive? But the first question was just
28:17
are we able to get the users to set up and answer any questions
28:20
by themselves? And so I think that's kind of
28:23
always the push down, like it's always a balance.
28:26
And I think a lot of the reasons why many AI products are not the
28:30
most intuitive is because it's a it's a very tough challenge to
28:32
crack and it comes at the cost of other features.
28:35
It's almost like you have to optimize for the early adopters
28:38
right now. Yeah, you do.
28:39
And and in a way you you put a lot of requirements on the user.
28:47
And so there is the balance of where, where do you do that and
28:50
where not. But it's also in the hope and
28:53
expectation that the users do spend time on learning the
28:56
system. Yeah, because one of the
28:58
challenges I heard was that folks would give up after trying
29:05
to do something or asking Tokan to do something for it, and then
29:10
it wouldn't do it the first time.
29:12
They would try again. But you don't really know.
29:16
Am I not able to ask this correctly or is this just not
29:20
possible? Yeah, yeah.
29:22
So that's super difficult to navigate as a user.
29:25
And there's a couple of things we've done around that for user
29:28
experience. One is just make the system much
29:31
more resilient itself. So when we build the first data
29:34
analysis, not the date, not the SQL connection, but just data,
29:38
network code, execution functionality and token, it
29:42
would fail almost all the time because it couldn't load the
29:45
file that you shared, or it couldn't handle the formatting,
29:47
or it would time out because it installed some absurd library.
29:51
And you as a user would never know.
29:53
This is something that you can't easily.
29:55
I mean, you could theoretically solving a prompt so.
29:57
You're thinking, oh, it's maybe because I didn't ask it the
30:00
right way. Right.
30:01
Or like, oh, maybe this is not even enabled right?
30:04
I've seen a video of it, but maybe that's something
30:06
different. Maybe I misunderstood It's.
30:08
Very easy to. I'm not using the newest version
30:10
of Tokon, but. Right.
30:11
And so it's for as a user perspective, these things can be
30:14
hard to interpret. So what we made sure is that
30:17
your first touch point is a lot better.
30:20
And so that that by itself actually helps a lot.
30:24
And just in terms of how the system responds, both on being
30:28
able to succeed, but also when it fails, like, hey, I'm sorry I
30:31
couldn't do this, but here's a set of things that I can do.
30:35
So your your first touch point being like instead of ask me
30:39
anything it is ask me this because I can't.
30:43
Do that. So that's the second point.
30:44
And that's more in the onboarding of users being like,
30:46
OK, well, here's a checklist of things we want you to try
30:50
because we know that these are generally valuable as
30:53
capabilities. And so some of these might be
30:56
really interesting for you, but it gives you a good ground of
30:59
some successes. And that's more on the
31:01
onboarding side of, let's say here's a list of 10 things.
31:03
So, and we can go over how we communicate with users in a
31:07
second as well, but summarize an article, right?
31:11
And it's like, oh, well, it can do that.
31:13
OK, that's one win. And even if the article isn't
31:16
relevant now, you as a user have one win.
31:18
It's not that you asked the first question.
31:20
It fails. You would try it again, it fails
31:22
and you're like, well, I give up, right?
31:23
Yes. So it's that.
31:25
But then, but then on the product side really or more on
31:27
the technical side, making it much more reliable, catching a
31:30
lot of the edge cases that could go wrong in order to help the
31:33
user, especially when they're when they're actually starting
31:36
their own work, not immediately get frustrated.
31:39
Now as you're looking forward, I know that you have thought a lot
31:43
about what to do next. What's the general consensus on
31:48
like how you're going to continue the progress?
31:51
Yeah. So right now we are very much A
31:55
and we're, we're in very much in the AI assistant category, a
31:59
chat interface that allows you to kind of have conversations
32:02
with AI, use tools and and complete tasks.
32:05
What we're looking at over the next 5-6 months is a step change
32:10
to our current paradigm is AI workforce.
32:13
So we can talk a bit about what that means, but the fundamental
32:16
question we've started to ask ourselves and there's hypotheses
32:19
baked into this, but as a platform in process, because we
32:22
are a horizontal solution, how do we allow every employee in
32:27
process to build AI employees that can complete their work?
32:31
My fundamental belief or so there is an observation and
32:35
these terms are flying around a lot AI employee, especially with
32:39
current technology. I think it's mostly a marketing
32:42
term, right? It's something that you used to
32:43
sell it to a stakeholder or something that you used to sell
32:46
it to your investors. AI employees as such don't exist
32:51
because we as employees are actually very complex situations
32:56
that I think in an ideal world you wouldn't want to replicate.
32:59
If I think of hiring and an actual employee, then hiring is
33:02
one of the most impact, imperfect workflows there is,
33:05
right? Because I have a team of seven
33:07
people, I need to wait until I have work for at least 8 before
33:10
I hire someone. And now I need to find the
33:12
person with the right skills to do all of that work that's piled
33:15
up. In my ideal world, I just look
33:18
at that work as jobs to be done right and very much in the
33:21
idealistic product, thinking of if I hire an analyst, one of his
33:25
jobs is cleaning data, one of his jobs is to generate weekly
33:29
reports. And there's many, many more
33:30
things he does. But in between, but, but there's
33:32
some chunks of jobs to be done if we look at it from that
33:36
perspective. And I believe that almost all
33:38
jobs you can break down into at least many of those, there's
33:42
going to be some fillers in between, but jobs to be done and
33:45
we actually take a look at jobs to be done and we try to do that
33:48
categorically. Then almost every job to be done
33:52
starts and ends either with another job to be done employee
33:56
or in a system. And so a lot of what we are
33:58
doing when we're completing our jobs is we create transformer,
34:02
move information around between systems.
34:05
And just to make that more practical, we can, we can look
34:08
at something like I might store my user research in Google
34:12
Drive. So I'll, I'll jump on a Zoom
34:14
call. There's a call, of course, that,
34:16
that one at the moment doing, of course, myself from that
34:20
interview, I, I get a transcript and a recording and notes.
34:23
And so those are already kind of piece of information that are
34:26
stored in the system. If I now wanted to get insights
34:28
from those notes, I take those notes.
34:30
I can either manually myself or currently you do put in some
34:33
sort of AI system, extract some insights and those insights
34:36
either directly go on the product backlog or they go into
34:40
another place and maybe in Confluence somewhere else.
34:43
And you can break this down for almost every job, right?
34:46
So let's take marketing. You have your campaign data
34:51
somewhere in, let's say Google Analytics or somewhere in Google
34:55
Adwords, in any place that manages your paid ads campaigns.
34:59
You take that campaign data, you analyze it to see what campaigns
35:02
are doing well. And then from that campaign or
35:06
from that analysis, you create again a file in Google Drive,
35:11
probably Slides, or if you use SharePoint, could be micro,
35:14
could be PowerPoint. Maybe a look or dashboard?
35:16
Something like that, Yeah. And, and you would use that,
35:19
but, but I think the important part here is we're moving it
35:21
into a report that we now give to someone else, right?
35:25
And that other person now looks at the report, comes up with a
35:27
set of recommendations, which could be in an e-mail in another
35:30
system that we then send back to the person that updates the
35:34
campaigns in Google, Google Ads, OK.
35:37
And so in all of these processes, most of the things
35:40
we've done is we've stored or moved information around in
35:43
systems and of course, we transform it along the way.
35:46
And so the way we look at the work right now or at the kind of
35:49
AI workforce paradigm is in an ideal world, not probably not
35:54
the next 5-6 months, but an ideal world.
35:56
We see that companies run drastically different with an AI
36:00
workforce because the core components are, they're kind of
36:03
three core components. 1 is the systems that we can use today
36:06
like your Salesforce, your looker, your, your actual, your
36:09
own code bases, GitHub, etcetera, the agents or AI
36:13
employees that are able to complete tasks in between these
36:17
systems. And so when we say employees,
36:19
again, it's probably more on a jobs to be done basis.
36:22
They might grow in scale, but but these are basically
36:25
connections between the systems in order to fulfill tasks that
36:28
need to happen. And then the third part, and
36:31
that's the architect role, is you'll probably get a subset of
36:36
extremely talented domain experts who are able to, instead
36:39
of doing the work, they think of how should my marketing
36:44
department work? What are the different elements
36:46
that need to be in place? And so they're going to be much
36:49
more in the process of building these systems, building these
36:52
employees, monitoring them, making sure that they're up to
36:55
par and that they're completing the tasks properly.
36:58
But it becomes much more of a machine that they're kind of
37:01
designing and running rather than doing the tasks themselves.
37:06
So I have to understand my field deeply, but then I also have to
37:11
understand how and where I can plug in the assistance in a way
37:17
it reminds me of. Like how is this different than
37:20
Zapier? Yeah.
37:21
So I think Zapier is difficult. So, so that's it's, it's the
37:27
right way of thinking about it because it is connecting
37:30
systems. And let's just assume Zapier is
37:33
a great tool and it works, which we can go down a whole different
37:36
tangent on how shitty that is, but let's.
37:39
Let's not make that the the topic.
37:42
Yes, I think you, you, you, you introduce a couple of new
37:46
elements. Zapier integrations, even for
37:49
technical people are quite difficult because you need to on
37:53
a on a hyper nuanced level, understand, understand every
37:56
connection part of the system. And so these connections are
38:00
very fragile. You need to deeply understand
38:02
them in order to get them right. They're very prone to break if
38:05
something changes. It's very difficult to build any
38:08
sort of almost conditional logic.
38:13
I don't want to say conditional logic because you can write your
38:15
own code in between, but if if it's not straight moving
38:17
information or always take this number and multiply it by two,
38:21
this kind of this cognitive aspect in between that currently
38:24
employees are doing that's borderline impossible to
38:27
replicate with Zapier. And I think the way we're
38:30
looking at the system now is as a platform, we need to provide
38:35
the architects with the ability to easily implement the
38:39
connections. And in part it's, it's easy
38:41
because if we abstract the way connections with tools, then
38:46
users should just be able to click write e-mail, right?
38:49
And they don't have to understand all the endpoints
38:51
that go into e-mail creation. It's just write e-mail as a tool
38:54
in that agent and update local dashboard.
38:58
And some of these are more complex and some are easier and
39:01
there's going to be a while for us to build up all the relevant
39:03
connections. But we as a platform should
39:05
abstract that. And then users should be not too
39:08
worried about the specific technical connections, but more
39:11
about the flow of information and the actual completion of the
39:13
task. And do you see it as something
39:15
that the user is going to have choices when they're working
39:20
with certain systems or tools that well, if it is Gmail, you
39:26
have choices of writing emails, adding people BCCI think we
39:31
have. That today actually.
39:32
So we or we're about to go live with our first set of
39:35
integration. So we've tried integrations in
39:37
the past. Users scream about and not we
39:39
can talk a bit about user discovery as well, but users
39:42
right now scream of, oh, we want Google Drive or let's, let's
39:46
stick with Gmail. We want Gmail and they're like,
39:48
OK, what does that mean? They're like, I don't know, but
39:50
I wanted right. It's it's very, it's very
39:53
abstract in many cases. And users know that that might
39:55
be interesting because that's where a lot of their workers
39:57
sort of they're paying us, but they don't know exactly what.
40:00
And what we're building now is a user level authentication and
40:05
integration system so that users can connect to their Gmail and
40:08
we now allow them to select the right the relevant tools.
40:13
And so the tools themselves have parameters were of course,
40:16
because it's basically a layer on the integration.
40:19
And so we're a bit dependent on the the what make, what the what
40:24
the integrations make available through endpoints.
40:28
But for the most part, agents get that information, as in this
40:32
is what the tool can do. And so they can ask the user for
40:34
questions and we can we'll build a learning overtime mechanism,
40:38
for example. Or you might always want your
40:40
manager CC D if you talk to this client.
40:42
Something like that. So the end user is going to have
40:45
a select set of things that it can do with the the different
40:49
services. But on the engineering team, on
40:53
your side, if you wanted to, you could build out a tool or build
40:58
out something that someone can do right.
41:01
Yeah, exactly. And and the way we also
41:05
communicate part of that to users right now is because let's
41:09
take, let's take GitHub. GitHub is probably a good
41:11
example an, an overload of endpoints, right?
41:16
So we're looking at 100 plus endpoints that the platform can
41:20
have. We've selected what we think is
41:22
the most relevant for now and build tools around that.
41:26
And I think we are ending up at around 20.
41:29
But even then that's something that users that isn't easy to
41:33
navigate, right? It's like, OK, well, you have
41:36
list repos, you have get PRS, you have get comments on PR, you
41:40
have make new comment, review PR and something else, right?
41:46
And so the way we're communicating at least part of
41:49
that to users as we've started to group them into skill groups
41:52
because we say in order for you to understand all of the nuances
41:56
of these tools, it's going to be somewhat overload for most
41:59
users. So we say, OK, if you want Token
42:01
to be able to manage your PRS, click connect here.
42:04
And it's going to automatically connect to six of these because
42:07
it needs to get access to your repo, it needs to be able to
42:10
read the comments, it needs to be able to add a new comment,
42:13
etcetera. And so we started to group
42:15
these. We'll adjust these along the
42:17
way, but this kind of framing around skills is quite helpful.
42:20
And then it's becoming a skill set, right?
42:22
This version of Token needs to be able to do these things in
42:25
order to complete my task. So just closing the loop on the
42:28
idea of the system and basically everyone needing or folks being
42:36
able to think. As an architect, you're now
42:42
thinking whatever your job is, you have certain repetitive
42:48
tasks in your day-to-day and let's figure out how we can
42:53
throw an agent in there to make your life easier.
42:58
That's the starting point, yeah. And yeah, sorry.
43:00
What's next? What's?
43:01
Next, well, we first need to get this to work, right.
43:03
So we're still in we're now in the loop of can we get this to
43:06
work at all Can we make it something that users can easily
43:11
set up and then can we make it so intuitive that it's going to
43:13
be a no brainer right? And so we're, we're still at the
43:16
start of making it work. We're about to release the first
43:19
wave of integrations. We already know that we need
43:22
some sort of memory or learning mechanism for this because even
43:26
things like tool call details. I'll give you a very, very
43:28
simple example that I've been using to illustrate this.
43:30
But right now, if I ask token, please schedule a meeting with
43:32
my manager, I get a sure, who's your manager?
43:35
Even if I've done the same thing 10 times.
43:37
And then I'll be like, oh, it's Yanis.
43:39
And then it's like, OK, well, what's the honest e-mail?
43:41
I'm like, it's the same thing every time, right?
43:44
And so it's actually just a very, very simple example of
43:47
something that it's actually just one parameter in the tool
43:50
called. But without this parameter,
43:51
we're not going to be able to do anything.
43:53
And if the ambition is to do complex multi step tasks between
43:56
systems and we can't get simple parameters right, then we're not
43:59
going to be able to do it. But I think this is an
44:02
interesting point where we look at memory now actually quite
44:04
differently than I think many other platforms, especially if
44:08
you look at how ChatGPT, etcetera, do memory, it's more
44:11
personalization. And we're looking at how can we
44:14
learn to do tasks more consistently and in a better way
44:18
by figuring out what the right inputs and outputs for tool
44:21
calls should be, How users are screaming for multi agent,
44:24
they're screaming for observability and evaluations on
44:29
these kind of more complex systems, etcetera.
44:32
And then, yeah, more beginner users are looking for
44:35
marketplaces, right? They're like, I don't know how
44:37
to start. I want to see someone else share
44:39
it and then I can maybe copy and update it.
44:41
Which is what I thought. I'm obviously the lowest common
44:44
denominator because I thought, man, when I go to Tokon, I would
44:47
love to see, hey, you're in XYZ job.
44:50
Here's a few great flows that you might want to get started
44:53
started with. We can connect this system with
44:55
this system and then do this small task for you.
44:58
Yeah. And that's coming when we have
45:01
those connections and we know they work.
45:04
It's been something we've steered away from before because
45:07
it's very difficult to share something that's actually
45:09
relevant to you. I mean it's very easy to to
45:11
overload you and things that someone in your kind of role has
45:14
done. But because every employees work
45:16
flows can be so different and that's also why we think that
45:20
agent creation needs to be bottom up for the most part, it
45:23
can be quite difficult to beyond examples share.
45:26
Hey, this is relevant for you because someone might work in
45:30
Tableau, you might work in Looker, they might connect to a
45:34
a specific database and you get your data fed from finance
45:37
through an Excel sheet. Like your workflows, your ways
45:40
of working are going to be drastically different.
45:42
And this is probably one of the more similar kind of flows.
45:44
And so sharing has been, here's an interesting example in where
45:48
we make trade-offs in the product we we've built or one,
45:53
one of the bigger things we've built over the last quarter is a
45:56
configurable version of token. The assumption was if we allow
46:00
users to configure a token for their specific task, not just
46:04
the token that they have available by default and they
46:06
can give it permanent knowledge, they can give it instructions,
46:09
etcetera. Then they're going to be able to
46:10
build a more powerful version that even without integrations
46:13
starts to be able to do some of those tasks.
46:16
And while that's true, one of the things that we've not build
46:19
out far is sharing. And so we thought, well, we will
46:24
build out sharing when we know that this configurable version
46:28
of Token is valuable. We know right now it's still
46:30
hard to set up. There isn't a lot of them that
46:33
are easy to you. There isn't a lot of them out
46:35
there that are providing a lot of value.
46:36
There's still a big gap in what users need.
46:38
And so building sharing now has no value because yeah, sure, you
46:43
have the ability to share, but there's nothing to share, right?
46:46
So make it work 1st. And so we have the simplest
46:49
version of sharing right now, which is a horrible user
46:51
experience, which is the admin clicks on share, they generate a
46:54
link and you can share the link with someone else.
46:56
And now you can join that custom version of token.
46:59
User flow wise it's not as far from ideal, but it works.
47:02
And it's a first iteration, I don't think you you don't need
47:06
to go that hard on yourself because it's but.
47:09
It's a but yeah, sure. But but it can be much better
47:11
for actually quite a little effort, right.
47:13
But it's the the concept of configurable versions of token
47:18
isn't where it needs to be at. So building out sharing, just
47:20
like the marketplace idea is, it sounds nice, but the reality of
47:25
it is probably going to be quite underwhelming.
47:26
In theory, because there's so much customization and that gets
47:31
to this other question that was going on in my head on why you
47:35
don't think about just verticalizing certain tools or
47:40
token products. I know that you thought all
47:43
right. Well with data analysts, we're
47:44
going to verticalize that. Yeah.
47:46
So, so it, it kind of just goes back to a bit of business
47:51
reality where because of our target audience and we are
47:56
somewhat limited in the addressable market in a way.
47:59
So we have 30 ish thousand employees, user experience
48:02
research. We have one big advantage in
48:04
terms of our users, which is that because they're all
48:08
portfolio companies, it's incredibly.
48:09
On the phone with you? It's incredibly easy to reach.
48:12
Well, they have to. They want to.
48:13
There's a lot more trust immediately.
48:15
There's a lot more willingness to spend time, I think.
48:17
You're not trying to sell them anything.
48:19
Yeah, exactly. We're there to listen to their
48:21
complaints. And I think when I started,
48:23
there was a lot of us reaching out.
48:25
At the moment, I might, my first agent might be something to
48:28
manage my inboxes because we get so many requests all the time
48:31
from all kinds of different teams.
48:32
And it's hard to manage, but it's, it's amazing in the sense
48:35
of seeing how many users are actually interested in this.
48:38
And, and the benefit we get from that is we get to take a step
48:41
back and abstract a bit of, well, OK, lots of people are
48:44
always asking for these things. I think this is kind of the
48:46
product management dream of, OK, we have 200 users from different
48:50
companies and different teams and different workforce and
48:52
different departments asking for something that we could
48:55
conceivably try to build like this, OK?
48:59
And then a lot of the, the user research balances 1 tricky part
49:04
is that, OK, we want to kind of build ahead for the future,
49:06
right? So we want to build new things
49:08
and that are barely possible or not really easily conceivable.
49:12
And so users are not really going to ask for that.
49:14
At the same time, users see all of this AI technology and
49:17
they're like, oh, I want an AI for my legal database.
49:21
And you're like, I don't know what that means, but if you ask
49:24
them, they also don't really know what it means They.
49:26
Just want to plug in AI because they know it's going to be
49:29
useful. And then you go to them and you
49:31
talk to them like, well, OK, when you say this, do you want
49:34
it to just have access to the documents?
49:36
Do you want to generate something for you like?
49:39
And then generation, should that work on a template or not?
49:42
They're like, I don't know. And you're like, well, OK,
49:44
that's, it's a, it's a very iterative process.
49:48
And I think one of the things we've learned is that we speak
49:51
to users a lot. And we have things like a Slack
49:54
channel that has 1300 users in there that if we have an idea,
49:58
we have a question, we just check it in there and we get
50:01
lots of feedback right away. But what we've learned is build
50:05
a first version as quickly as possible, like the integrations.
50:09
Our assumption is that the user level authentication on top of a
50:13
layer of APIs is good enough for many workflows.
50:16
We'll see. So we want to give that to users
50:19
as quickly as possible, see how they react, what they complain
50:22
about. And then if we need more, we'll
50:23
build more. If it's good enough, it's good
50:25
enough. What we know is that we need to
50:27
be able to talk to their systems.
50:28
So this is our first best guess at how we can do that in a way
50:32
that we can actually scale. How are you using Breakdown the
50:36
flow of how you are using Tokan in your own job.
50:43
So I have a, a flow that might make me, I think it's a Home
50:48
Office flow, but I'm, I'm in the office everyday.
50:50
So, and, and I'm saying that particularly because when I come
50:54
to work in the morning, the first thing I do is I bunker
50:57
down in a meeting room and I just use the voice input to just
51:01
get out all the thoughts of my head into structured list.
51:04
Wow, cool. It's just a, it's not a like
51:08
feature wise, it's not that it's not that impressive, but it's a,
51:12
it's a very interesting flow of just, I hate typing and I think
51:15
every user cuts corners and typing prompts.
51:18
But if I can just get out all my thoughts in one go, then I start
51:22
a lot of the things that I need that I know I need to get done
51:25
for the day. In the 1st 15 minutes of my day,
51:27
it might start 456 different conversations and all I'm doing
51:30
is I'm providing the relevant context by just to be babbling
51:35
on for 5 minutes. That creates a massive prompt
51:38
that I can then use as a base for tasks that I want to do down
51:41
the road. And this can, this can, this
51:45
depends on what I need to do, but this can be creating
51:47
presentations and then kind of reports and structures like
51:50
that. And this can be preparing for
51:54
webinars, doing some research, this can be doing a product
51:58
update. And so there's I, I still find
52:01
myself as my head being the best source for a lot of this data.
52:05
I can't easily point it just as a Google Docs, but it's very
52:08
easy for me to just summarize what's in the Google Doc and
52:10
provide the Google Doc. And then that's a great
52:13
foundation. And then, for example,
52:16
presentations. There isn't a single great tool
52:20
to build presentations I've seen all of them are.
52:23
I think my favorite so far has been Gamma.
52:26
Yeah, I've seen that as well. But I think it does, it falls
52:29
into the repetitive consulting style slides where it creates 3
52:35
columns instead of headers. And it's always it's, it's
52:38
always so far from something that I would want to use.
52:42
And especially if you want to explain something, it's not
52:45
easily able to visualize and make connections.
52:47
It's, it's more there to breakdown content and put it in
52:51
a slide that you could present, but it's not easily telling the
52:54
story. So now we've, I've just started
52:57
having to kind of create HTML reports.
52:59
And so now I'm going to have a road map session next week with
53:03
our users. And the plan is that I just have
53:05
an HTML page that I click through that I actually use as
53:08
the presentation, which isn't the presentation, but it works a
53:11
lot easier for token to generate it.
53:14
There's things like this data analysis.
53:15
I don't write a single line of code.
53:18
I'm banned from all of our code bases and making any technical
53:22
comments. So any sort of kind of review of
53:25
data is a lot easier for me like this than if I have to go to
53:28
Google Sheet or Excel. Thinking about how you, we were
53:33
talking earlier on the architect and what you would do when it
53:37
comes to you give this brain dump through a voice technology,
53:43
it then transcribes it. You fire it off to Proms.
53:46
You get that. How do you see that system
53:50
working in the jobs to be done and all these tasks?
53:54
Yeah. So, so I think there's two
53:56
things missing for me to lean into it and then eat her own dog
54:00
food, which is just that one. You can't yet configure a
54:03
version of Token that has integrations.
54:05
We'll have to add that soon. And so right now there's just
54:08
one token and I can't have, oh, this version of Token writes my
54:11
product updates and it has all the right sources.
54:13
And this version of Token and that's my user research and it
54:16
has all the right access. And so you see it as different
54:19
token agents like GPTS. I think so, yeah, it's a yeah,
54:23
GPTS different, different agents essentially because we know that
54:28
completing tasks especially like because we're not really talking
54:32
about read an e-mail and response, right.
54:34
But in an ideal world, we'll talk about do a product update
54:39
by looking at all the relevant things that that we've said on
54:42
Slack and look at all the things that have gone into get up, NP
54:46
Rs and look at the Jira board and then write it up as an
54:50
e-mail, send it to to people from the team to see if there's
54:55
anything else they want to add. And then put it together as
54:58
like, like a slack blocker that we can send out through a
55:00
messaging system. And, and that actually is still
55:03
a somewhat simple to help because it's not simple, but
55:05
it's a, it's a very clear step by step.
55:07
There isn't a lot of thinking in between.
55:09
If I talk more on strategic planning, for example, it's
55:11
like, well, OK, well, sure. I want you to know where to, OK,
55:14
I want you to find the latest product plan.
55:17
I want you to be able to get a current status of the product.
55:19
I want you to get the data for the product.
55:21
I want you to do outside research.
55:23
So these are much more complex tasks to combine and we know
55:27
that at least with current technologies and model
55:30
performance that becomes possible, but only if you have
55:33
an agent that is actually very tailor made to do that specific
55:36
thing. Otherwise it can go all over the
55:38
place. Consistency goes down, so it
55:40
might work two out of three times.
55:42
And so configure, configuring a version that is tailored for the
55:47
job to be done is important. And that actually allows you
55:50
later on to share it, right? Because if I just share my
55:53
token, it's going to be harder because the likelihood of
55:56
someone doing exactly the same jobs to be done that I'm doing
55:59
is somewhat low. But managing A backlog, every
56:02
product manager does. Doing user research, every
56:04
product manager does, right? And so that, yeah, makes it also
56:07
easier to scale. And then there's a version of
56:09
this where you share yours with me and I can just copy it and
56:14
swap out a few things because. Basically it's template you, you
56:18
can, well, there'll probably be two ways. 1 is just use it as
56:21
well. So just join, use it.
56:23
And that is probably more relevant for the company, like
56:25
within a company. So if I have, if I have a data
56:30
analyst in a company, the likelihood that the next data
56:32
analyst has a very similar systems and access and rules is
56:36
π if we talk about sharing across companies, it's probably
56:39
more like a template or a blueprint that you could update
56:42
and change out systems, change instructions.
56:45
When we talked to Flores last time, he mentioned there was a
56:48
hackathon you all did and the folks that won the hackathon
56:53
were a JIRA agent. But then everything worked
56:56
really nicely in the hackathon, but as soon as you plug it into
57:01
a real world JIRA scenario, the agent had no context that didn't
57:07
understand what was going on. How do you think about that with
57:10
the integrations so that it actually works?
57:13
Yeah, I think the the hackathon we did was, I think it was the
57:19
first time we did a hackathon around tool building.
57:21
So there's a lot of things we learned around tool building #1
57:24
is that if we build a tool assuming how a system works,
57:29
then things are not going to turn out well.
57:31
And so one of the problems is that if we build a JIRA agent
57:35
ourselves as a team, it's another reason not to
57:37
verticalize. And we just assume everybody
57:39
uses JIRA in the same way, or all the boards are the same, or
57:42
everybody has a clean backlog, or no, everybody uses labels,
57:45
whatever JIRA looks like, then, then this is not going to work.
57:50
And so our thinking is the core building block is access to
57:55
JIRA. And in the configuration you can
57:57
tell your version of token. OK, well, we just use, we use,
58:02
we just use Kanban. So there's one continuous Sprint
58:04
and there's just to do, refine, to do in progress, done.
58:08
And those are the only tickets that matter.
58:11
And whenever you create a ticket, follow these
58:13
instructions. So it's really about access to
58:16
the to the integration or it's really about the integration
58:21
itself, but scoping that integration and shaping it in
58:24
the way that you want to use that.
58:27
Yeah. And then and then the main part
58:28
there is what are the actions that users want to take in
58:31
systems, right. So if I if we go back to the
58:33
skills metaphor, then it's if I have my version of token, what's
58:38
the particular skill that that employee should have, right.
58:41
It should be able to create the ticket.
58:43
OK, well, and now it's more on contextually, how do you guide
58:46
it to do the right ticket in the right moment of time?
58:49
And that's another reason for this kind of knowledge learning,
58:51
right? It's a it's parameters in the
58:53
tool call saying this is the space we use in Jira.
58:56
This is the board we use in Jira.
58:58
And now you learn over time that when I say, hey, for the Tokan
59:02
team, can you please add an infrastructure ticket that this
59:05
that does XYZ. It's like, OK, well, I know we
59:08
don't use labels. We've learned that before.
59:10
I've made that mistake before. I just going to put a square
59:12
bracket infra in front of the ticket name and I'm going to put
59:17
it on to do with the same board that we use for everything else.
59:22
But this is this is learning over time and and this is also
59:24
our first draft, right. And there's things that we'll be
59:27
able to make easier. If I think on more like the two
59:30
year horizon and also at process at more like an investor
59:33
perspective. Then the really interesting part
59:36
comes in when we can say we can buy a company and within a month
59:40
we can start deploying an agent fleet across the organization.
59:44
And that probably means some crawling and indexing and
59:47
learning from existing data and not having to rely on only
59:49
users. So crawling and indexing, talk
59:53
to me. Well, that's an interesting
59:55
piece too. And it's fascinating to think
59:57
that you could do that. Learn from the systems that are
1:00:00
already in place. Learn how they set up their JIRA
1:00:04
and what kind of if they use tags or if they just use
1:00:07
brackets. So it's, it's something we've
1:00:10
deliberately not touched because, because looking at the
1:00:14
current set of hypotheses, we say, well, the first AI
1:00:17
employees are going to be bottom up.
1:00:19
And that inherently means bottom up and very, very narrow in
1:00:24
scope. And again, micro but and because
1:00:28
of the AI employees that we've been looking at as examples and
1:00:30
as use cases for use to build up the relevant context is almost
1:00:35
always hyper focused on just completing that task.
1:00:39
So when I do that, it has absolutely no, let's say I'm I'm
1:00:44
doing a flow somewhere in product.
1:00:48
It matters not at all that currently sales has an event
1:00:51
ongoing somewhere else in the company or the how an operations
1:00:55
team might structure their dashboards.
1:00:57
Absolutely no relevance to the specific job I'm doing.
1:00:59
So context in that sense is overrated as in company context
1:01:03
I think maybe I'll take, but the specific context is the most
1:01:07
important part for completing job.
1:01:09
Again, probably current assumption is that that's tool
1:01:11
call specific. We currently think that the best
1:01:17
way to get that out is through user interactions first because
1:01:21
we wouldn't know where to start when crawling.
1:01:23
I think crawling is such a huge topic and you can almost every
1:01:28
system is, I don't want to say poorly managed, but if you look
1:01:32
at any Google Drive, it's a mess.
1:01:33
And if you look at any Confluence, it's a mess.
1:01:36
And Jira and GitHub has whatever.
1:01:38
And I don't want to talk about Salesforce at all.
1:01:41
And like, so these are things that are very difficult.
1:01:44
And I think almost one of the things we're doing right now is
1:01:46
we build a golden test set that says, well, we've been talking
1:01:50
to this user for three months in order to complete this work.
1:01:54
And now it's running completely autonomously.
1:01:56
And these are the things we've learned.
1:01:57
Let's scrap that and say what happens when we crawl and index
1:02:02
the system? How can we learn something that
1:02:04
gets us close enough in order to say, OK, this is a good starting
1:02:08
point, but we don't have that test set, so it's not something
1:02:11
we're going to start, so. Basically crawling and indexing
1:02:14
just creates so much noise. Yeah, and, and maybe crawling
1:02:17
and indexing isn't the right way of looking at it.
1:02:19
It's really. Trying to understand.
1:02:22
Reading, reading and extracting, right?
1:02:24
It's you want to, you want to extract the relevant pieces of
1:02:28
knowledge. And I think right now we don't
1:02:29
know the rules for that. It goes back to there's certain
1:02:33
paths that are taken, there's certain signposts, and if you
1:02:37
think about the 8020 principle, there's probably 20% of the
1:02:42
stuff that you're doing on JIRA is very important.
1:02:45
And you know that, yeah, But the other 80% is still there.
1:02:51
And so if you have everything weighted the same, it's going to
1:02:54
create a horrible system for the agents.
1:02:56
Yeah. And users might work in
1:02:57
different ways. And so then do we ignore it?
1:02:59
Do we take one? I think there's another actually
1:03:01
really important point that that I skipped at some point, which
1:03:04
is many of these agents will require the user to redefine how
1:03:08
they currently work. My, my current, and again, on a
1:03:13
very specific process level, my current way of writing product
1:03:16
updates is incredibly chaotic. It's very messy.
1:03:18
It's not a process at all. And the reason is I don't need a
1:03:21
process. I know where I need to go when I
1:03:23
do it. I go there, I go here, I look
1:03:25
at, I talk to the team and I get it done.
1:03:26
But that's not agent friendly. It's not going to work for an
1:03:29
agent and it might not be best, right?
1:03:30
This is very much human nature thing of like I mean some some
1:03:35
product managers would have built a process around it.
1:03:37
I'm more non process so I wouldn't have done it in the
1:03:39
first place and I'm going to not do it until I have to hand it
1:03:42
over probably. But so when we talk on the
1:03:47
learning from existing knowledge, we basically just
1:03:49
look at existing processes. We don't have any space to
1:03:51
redefine these, right? And so very much the architect
1:03:54
comes back into place and saying, well, maybe the right
1:03:57
thing to do is of course we can look at what's there.
1:03:59
But maybe what we should do is say, OK, this looks like what it
1:04:02
is. This looks like it should be the
1:04:04
jobs to be done rather than. This is the rules to replicate
1:04:06
exactly what you're doing right now, because they're replicating
1:04:09
exactly is I think, more likely to not work than giving a
1:04:14
suggestion of how you could refactor the process work right?
1:04:17
I think that's maybe an interesting view on this.
1:04:20
So the reading and learning and extracting knowledge inherently
1:04:25
takes all the existing things as gold standard, or at least in a
1:04:29
in a first version, which might also not be a good way to go.
1:04:33
That's all we've got for today, but the good news is there are
1:04:36
10 other episodes in this series that I'm doing with Process deep
1:04:40
diving into how they are approaching building AI
1:04:46
products. You can check it out in the show
1:04:49
notes. I leave a link.