Latest / Agentic Conversations (formally mlops.community) / Building Effective Products with GenAI // Faizaan Charania // #187

Transcript

Building Effective Products with GenAI // Faizaan Charania // #187 · Agentic Conversations (formally mlops.community)

0:00
Hey everyone, my name is Faizan Charania, I am a senior Product
0:04
manager at Linton and my favourite coffee.
0:08
Actually I am trying to decaffeinate myself so I'm
0:11
actually trying to not consume copy right now.
0:14
So my favourite coffee right now is water, which is, but that's
0:20
it, it's. Welcome back to another Mill
0:24
Office Community Podcast. I'm your host, Dimitrios, and
0:27
today we're talking with Jason. Wow.
0:29
I love this conversation because it is not every day that I get
0:33
to talk to product to people. But when I do, you know, I go
0:36
wild. I'm one of those guys that has
0:39
been talking to engineers for the last three years and seeing
0:44
the odd product person come through every once in a while
0:48
just it lights me up. I love it.
0:51
And today was no different because today we dove into how
0:56
to think about building a I products.
0:59
You've heard us talk about it before with other product
1:02
managers and this time what? It was so nice to hear how he
1:10
thought about. Generative AI as a tool in the
1:13
toolkit, not some hammer and go around looking for a nail, which
1:18
is common sense by this point. But I know I have seen the
1:22
products out there, especially products and features being
1:25
incorporated into products that look like they're just trying to
1:30
hold on some Jenny I or AI capabilities because they have
1:35
to or they're looking to make their investors happy now.
1:38
He goes into the different ways that he has found gotchas while.
1:45
Putting AI into. The different products that he's
1:50
been creating, we didn't get to dive in too much to what he's
1:54
doing at LinkedIn because for that we would need some PR
1:58
permissions and we didn't get it, so we brushed over that
2:02
whole thing. So if you're looking for his
2:04
full on experience at LinkedIn and what he's doing with AI
2:08
there, that will be for a whole another episode where we get the
2:12
permission. For this one, however, we talked
2:15
about. Troubleshooting we talked about
2:19
incorporating your AI features. We also talked about how he has
2:25
a. Product Manager speaks with and
2:28
dives into the technical aspects of the platform.
2:32
Team O hope you like it. This is, of course the Milos
2:38
Community Podcast, and we'd love it when you share this with one
2:42
other person that you think will find it interesting.
2:46
All right, let's get into it. All right, I got to start with
2:55
this dude. You are probably the first
2:58
person to come here. And have a past life as an
3:03
actor. Tell us about that, man.
3:07
Yeah. Yeah.
3:08
So this was when I was 5, actually, so I could start
3:16
eight. Yeah.
3:17
It's as I was by, I think I was out for some summer activity
3:21
somewhere, and there was this one shoot happening for some
3:26
commercial. And then I was just standing
3:28
there. I was looking at it and my dad
3:30
ended up speaking with someone and they were like, hey your
3:33
kid, do you want to get in, try out, audition for things.
3:37
And my dad was like might as well.
3:39
It's a good learning experience and it started with that And
3:42
they got two. I have a whole portfolio and I'm
3:45
doing auditions and I ended up acting in some Bollywood movies.
3:49
There was a prime time sitcom and we did like 350 episodes of
3:54
it and it didn't rise. And whether the age of 14 and I
3:59
was like, OK, I am bored of this now And they wanted me to
4:04
retire. Yeah, yeah, yeah.
4:06
It was either school or acting. So I was like, OK, now I'll just
4:09
stick with school. You said I see a big future in
4:13
machine learning. I'm going to chase that dream.
4:16
Yeah, yeah, yeah. Ohh dude that is so wild.
4:21
So anything that I may have seen your Bollywood days, I don't
4:28
know how many body movements you've seen, but then the movies
4:31
that I've done are a bit one of the top production houses.
4:34
So they're like out there, they're popular.
4:36
I don't actively advertise the actual movie names, not putting
4:39
it out on the Internet. We gotta go find your IMDb page.
4:42
All right, we'll post that in the show notes in case anyone
4:45
wants to check it out. It's classic dude that is so
4:50
good. Were you doing the whole dancing
4:51
and everything? The whole 9 yards.
4:53
Yeah, yeah, absolutely. I love dancing.
4:55
I still dance. Ohh man that is awesome.
4:59
Alright, that's that's my opinion.
5:00
But my mother is not working and let me just go ahead and dance
5:03
somewhere. When we do the in person ML OBS
5:07
Community Conference, we may have to have you be one of the
5:13
intermissions. You're you, we can me and you
5:15
both. I'll do it with you.
5:17
We'll create a dance routine and that's that.
5:21
OK, let's do that. OK.
5:23
So we didn't come here only to talk about Bollywood.
5:25
I know that that was a bit of a tangent start.
5:29
This isn't a retired Bollywood. Where are they now?
5:32
VH1 behind the music type thing. This is a talk about you being a
5:38
product owner and getting into the wonderful world of MLM's.
5:43
And I especially want to talk about the evolution that you've
5:46
seen because you've been in the game for 8 plus years.
5:49
So before you were what people probably called ML product
5:54
leader and now you're like an AI product leader.
5:57
You got to. Hopefully you got a pay raise
5:59
with the. Of new title.
6:00
But the cool thing is I would love to just start and hear your
6:05
story after the Bollywood fizz, then after school.
6:09
You obviously did well enough in school to get yourself a job.
6:13
You came to the US at some point and you started working for
6:16
Yahoo. And what happened before Yahoo,
6:21
I was always involved in they actually even at Yahoo to some
6:25
extent. I was involved in the very
6:27
technical aspects of machine learning.
6:29
I started out as a research analyst.
6:31
I ended up working with research labs.
6:35
I worked with some investment banks and their resource
6:38
departments. And then I did grad school
6:41
focused on machine learning, took some grad PhD level courses
6:45
that I didn't really bad at. But then I joined Yahoo as a
6:50
machine learning engineer, and a few months into Yahoo, I
6:53
realised there was this alternate troll.
6:55
Which was a product manager, like a machine learning product
6:58
manager. And they're like, hey, this
7:00
person has to be technical for them to keep their job.
7:04
But then they are also not very actively just working on at the
7:08
lower level, like where they're actually executing and looking
7:11
in GPU memory capacity and things like that.
7:13
But they're still in modern machine learning and they're
7:15
solving a lot of different kind of interesting problems.
7:18
So at some point I made that decision like, hey, OK, both of
7:21
these parts look very interesting.
7:22
I cannot actually pursue both of them.
7:24
I have to choose and I ended up choosing the product that's been
7:27
part, but it's coming from a technical background and also
7:30
being an MLM IT I've had to be extremely technical the whole
7:35
time. So that's something that I
7:37
really enjoy. And so yes, Machine learning PM
7:40
at Yahoo. And then I ended up moving to
7:42
LinkedIn and now our PMO here. Awesome.
7:45
Yeah. And I want to get into a little
7:47
bit of what you're doing at LinkedIn and if you are the one
7:50
responsible for the, when I post on LinkedIn the AI feature.
7:55
But before we get into that, you mentioned something interesting
7:58
there, which is how you have to be technical, but you're not at
8:01
that lower level. What are the technical pieces
8:04
that you feel like? You definitely need to know
8:06
about the technical pieces that I so generally when I'm trying
8:12
to build, say, any kind of machine learning product, I have
8:15
to understand how it's going to work under the hood.
8:17
Otherwise, it's just going to be random ideas and my engineers
8:21
will laugh me out of the room. So when I'm trying to think of a
8:25
user problem, I'm thinking of a user problem.
8:27
I'm thinking of, OK, how can we solve this using machine
8:30
learning? And then once I've started going
8:33
down that path, I'm thinking, OK, what kind of data do we
8:35
need? What kind of data do we actually
8:37
have already? Can we leverage that?
8:40
How can we leverage that? And what kind of experience is
8:42
needed? With other PMS I've seen
8:46
sometimes there's a very clear cut handoff between the product
8:49
idea and like how we end up building it.
8:52
The working model that I have with my team is I start at the
8:55
highest level and then I keep adding details and I keep adding
8:58
details. And then there's a section where
9:00
me and my tech leads, we are actively working on defining
9:04
what the solution will be. And at one point, I just hand it
9:07
off to them, OK. Now it's going to be much more
9:10
about what the implementation should look like and we don't
9:14
need any more product positions right now.
9:17
So they do that and if they have any product questions, they come
9:20
back to me later. So does that mean that you're
9:25
really intimate with? The types of technology your
9:31
teams are using, are you building out diagrams and
9:34
figuring out, OK, we're going to be pulling data from this
9:36
database and then making sure that we have all this type of
9:42
whatever, we're throwing it into Kafka Stream or we're getting it
9:45
from a Kafka stream and they were transforming it with Flank
9:48
and that kind of thing? Or is it more just like hey this
9:52
is our problem, I know that we have the data because I know.
9:58
From this other project that we created, we definitely are using
10:02
some similar data and I know that a similar project that
10:06
we've created also had. Success.
10:10
So we can do this like break it down, how does that look and how
10:14
do you also think about those things and think those things
10:17
through and talk to your team about it?
10:19
Yeah, it's one of the latter. As much as I enjoy getting into
10:22
the meat of like Kafka stream it is coming from, I don't think
10:27
that's the best use of my time and that is why I have these
10:31
brilliant engineers working with me.
10:33
So what I do is I focus on the latter part, which is OK, what
10:36
kind of features do we have? How can we use these features?
10:40
Sometimes the the one step extra is I know what data we have
10:45
available, what kind of experience that I'm trying to
10:47
build and then I'll have recommendations of like, hey,
10:50
maybe I don't think we can build a classifier at this point.
10:53
Maybe we can just do this heuristic and then we collect
10:57
any data and go from there. Or maybe we just start with some
11:00
low form of clustering and maybe we just use these attributes of
11:04
a user to create the MVP experience and then go from
11:08
there. One thing just like came to me
11:10
while I was talking about is the reason why this is potentially
11:14
useful as well, just simplifying the solution even when generally
11:19
the solution is owned by the engineers.
11:21
Theoretically speaking is because whenever we are building
11:24
an AI product, it's you cannot build the best model right away,
11:29
you have to start smart. Otherwise it's just a lot of
11:32
effort for confirming or even rejecting some hypothesis.
11:36
O what we do is I try to simplify the MVP as much as
11:39
possible. See if you're getting the right
11:41
signal. See if we are accepting or
11:44
rejecting the right kind of assumptions and then once we
11:47
have that, then we need a good machine learning model.
11:50
And then I take my hands off like, hey, OK, ports and
11:54
airports and we just build me a nice thing and then they own the
11:58
whole thing and I don't have to know any details.
12:00
So the interesting thing that you said there is, for me at
12:04
least. You're trying to figure out at
12:07
what? Basically.
12:10
Scale. You need to implement machine
12:13
learning and how, if at all, machine learning or AI in quotes
12:19
needs to be used because it feels like.
12:23
You're going for speed and you're trying to say, alright,
12:26
for this feature, let's see if we can get something U.
12:28
Let's see if we can get something going.
12:30
And then you're looking at the metrics and you're looking at
12:33
what is important when it comes to that feature.
12:36
And if there are some significant changes then you
12:39
throw you layer on ML on top. Are there times where you jump
12:43
from nothing to whatever the hardest thing that you could
12:48
possibly implement because you know it's going to work, or is
12:51
it always a gradual? Implementation and generally I
12:57
try to establish that I start with a casual implementation
13:00
always because there are going to be assumption.
13:04
If I'm building something new, there are assumptions baked in,
13:08
so generally I try to do a very simple thing first and then
13:12
build on top of that. But yes, there are going to be
13:14
situations where we have done something similar in the past or
13:17
we have valid data-driven proof that this absolutely works.
13:22
Now in that case we can tune up the complexity and then then I
13:27
work with my team to see when can we ship something by how
13:31
complicated does it have to be and work from there.
13:35
How often do you actually ship something on time?
13:39
Like, how often do you correctly?
13:42
Analyse how long it's going to take you to build some kind of
13:46
machine learning array AI feature and there's no more
13:52
often than not actually. And I I've been the engineer on
13:56
the other side and I know that's extremely difficult but then the
13:59
partners that I'm working with that absolutely brilliant.
14:01
There are obviously like something like some other things
14:04
that throw us off, but most of the time we are shipping on
14:07
time, so they've gone for something.
14:10
Kudos to you guys. So there there is something
14:13
else. I mean when you talk about
14:16
building these features and and scoping them out and thinking
14:20
through them. You had mentioned to me before
14:25
we hit record that you wanted to jump into a little bit of like.
14:31
What founders and builders should think about when they're
14:34
bringing generative AI into their products.
14:38
And I know that you have been. Really thinking deeply about
14:41
this and I love the stuff that you share on LinkedIn and I love
14:45
just like your mentality around it.
14:47
So I want to hear what's your thesis or MO when it comes to
14:54
building products with this new type of AI features in it as
14:58
opposed to what you saw before where it was it was mainly ML.
15:04
OK, if I try talking about this, I could just keep speaking and
15:09
you won't get a chance. I get it.
15:10
But to to keep it extremely short and the answer is kind of
15:16
counterintuitive to some people is hey, it's the same, there is
15:20
no change. We have to think about
15:23
generative AI as a tool and nothing more.
15:27
See how this tool can be used in service of your users?
15:33
So if I had to put it in one sentence, I would just say hey
15:36
Jenny is just a tool, you're still serving the users.
15:39
Build what you can best to serve the users.
15:43
Oh, I like that a lot. What are some key areas and ways
15:47
that you feel like we can incorporate generative AI into
15:50
our products? Indeed?
15:54
And again, this answer has changed every like one or two
15:58
months in the past year. Every open AI update, every
16:02
opening I have it, or like every open source model update, Even
16:07
when Lama Two came out, everyone was like, OK, now I can actually
16:10
use these services. But anyway, OK, so the first
16:16
thing in the simplest version of things, and if someone's not
16:18
trying to add too much complexity to their product,
16:21
what they can do is use LM's for creation of anything.
16:28
And mostly like starting with text, because that's easier.
16:32
Do this when you want to create some human aspect or not.
16:38
Make it very robot, not not make your experience, but robotic
16:41
when you want to add that sum and touch, which ironically is a
16:46
start with there. And once you start adding more
16:50
complexity or product, you can start addressing more problems
16:55
that need factual information. And again, anyone who is
16:59
listening to this podcast knows about hallucination.
17:01
So you have to actively fight against hallucination,
17:04
hallucination. So you start with experiences.
17:07
The easiest experiences to build are the ones that don't care as
17:10
much about the hallucination. And then you keep starting, keep
17:13
adding complexity, and then you can add the trivial augmentation
17:16
Commission. No, no.
17:17
You're basing your outputs and truth, so you can start working
17:20
on those things. Again, I think this is a very
17:23
technical explanation, but if you if I had to think about OK,
17:27
what kind of experiences are the highest value, I think that
17:32
depends from one company to another company. 1 product to
17:35
other product assistants are not for everyone.
17:38
I know it's very easy to build, but I feel like again this is a
17:43
very personal and I feel like a lot of people should not start
17:47
building assistance and they they yes hot take actually and
17:53
build experiences in your existing product and solve the
17:58
hardest problem. Because the assistant product
18:03
cycle requires you to request the customer user to talk, and
18:07
it's a pull mechanism and that's a lot of effort talking to
18:11
someone to get value. How about we cut through the
18:14
chase and provide value using generative aid?
18:19
Ohh, so talk to me a little bit more about this pool mechanism
18:22
as opposed to a a push mechanism I guess.
18:25
What what would a push mechanism be?
18:28
So push mechanism would be? Let me think about non LinkedIn
18:33
use cases. To not have to go through the PR
18:38
people exactly, it's something that potentially.
18:41
So let me let me just take a very simple example and this is
18:45
a public product that's out there.
18:46
This is the Help me Write section that you would see on
18:50
Google. Is it at Gmail?
18:52
I think they're testing and it's in beta right now, but that
18:57
there's still one level of interaction where they're
19:00
saying, OK, you want to write an e-mail, tell us what you want to
19:04
write about and they'll create an output for you.
19:07
So yes, it feels like I am telling the model to what to
19:11
write and then it's getting an output.
19:12
But there's so much happening in the background and all of that
19:16
complexity is removed from the user.
19:19
They're trying to explicitly Create an e-mail.
19:21
You can write something very simple in there and it will
19:25
still create a good e-mail from it so that there are meta
19:29
prompts involved. There are probably looking at
19:31
what your contact list is, they're looking at what the
19:33
thread is, thread of the conversation is, and all of
19:37
those things you don't have to worry about because it's already
19:40
baked in. That complexity is abstracted
19:42
away from you. So building those experiences
19:45
where you do the heavy lifting, the product does the heavy
19:48
lifting, and the user has to do minimal work, That's where the
19:52
most value is going to be. And the counter example of this
19:55
would be I take my problem to charge PT and say hey write me
20:01
an e-mail and now it's actually doesn't know what e-mail where
20:04
are you, where you writing to? What is the context?
20:07
What went? Wrong.
20:09
Yeah, yeah, yeah, exactly. It's like having to basically
20:13
abstracting away all of the prompts and the context and the
20:16
knowledge base that you need, right.
20:18
It's just, oh, it's going to seamlessly give you something.
20:22
And I understand now why it's like that push example.
20:26
It's going to push you the information that you're looking
20:29
for as opposed to you having to like pull it out of it.
20:34
Yeah, yeah, yeah, that's cool. Yeah.
20:37
And I think another one that is doing this well now that you
20:40
mention it, I can see Notion being one of those where when
20:45
you're just clicking around you don't have to explicitly add
20:50
special prompts unless you want to.
20:52
And to be honest I tend to find that when I do try and throw in
20:58
Proms for like write me something, write me XYZ with
21:02
whatever and notion I end U just getting frustrated.
21:05
But when I have something there already and it's like, hey,
21:08
clean this up or make this better, make this sound nicer,
21:11
punch it up, then it works quite well And and that's just
21:15
clicking a few buttons as opposed to adding some prompts
21:18
and so I can see that for sure. Also this notion.
21:24
Experience and even the Googles help me read experience.
21:26
They meet you where you are right now.
21:29
Yeah, I am writing this e-mail in this box and I can work with
21:33
the AI over here. Rather than you going to a
21:36
different vendor or a different assistant that's out there in
21:39
the same window and then having a conversation over there and
21:41
then putting it back, which is again, a lot of work.
21:45
Yeah, totally, totally. You did say something earlier
21:48
around when the open source models come out and how you felt
21:54
like people were excited about Llama Two dropping and it's Ohh
21:58
now I can use this. I imagine at an enterprise level
22:02
you're looking at things differently.
22:05
We don't have to talk about that at all, to not have to go
22:07
through PR or whatever. But how did you see the
22:12
difficulties in your own use cases or your friends use cases
22:17
of using things like chat, CPT or Open AI and it being super
22:26
expensive? And then also same goes for
22:29
like, I guess anthropic is another one that can be really
22:32
expensive. And we've talked at length on
22:35
this podcast just for a little bit of context about how there's
22:39
kind of two modalities of thinking right.
22:41
One is that I'm going to get something U and running as
22:47
quickly as I can to have that value for the end user and then
22:53
once it gets to a certain point that we can see this product
22:58
feature is useful, then we'll switch.
23:01
But until we get there, we're not going to switch and
23:05
potentially there's a lot of low hanging fruit that we can tackle
23:08
with the LMS. And if all we need to do is hook up to the Open AI
23:15
API, then we're good. And so they don't necessarily
23:19
think about that the expense wise.
23:21
But then there's a different category or a different school
23:24
of thought where it's no, I don't want Open AI to touch my
23:27
data and I don't. Or maybe it's not even that.
23:30
It's just like, yeah, that's really expensive and I can do it
23:33
myself, although I have thoughts on that that we can get into in
23:37
a minute. But they prefer to use open
23:39
source, and some people are just really interested in open source
23:42
because it's open source. So talk to me about how you
23:46
think through those things. Yeah, yeah.
23:49
Actually, we have come to the open source part in a minute, so
23:52
let me just address the. How do we approach, how do I
23:56
approach thinking about these solutions?
23:58
I am very much in the first cohort that you described.
24:03
I think LMS are so useful to increase the go to market speed
24:08
for anything that we are building.
24:11
Having said that, it's it's not going to scale to 100 million
24:16
users. It's going to be extremely
24:17
expensive, but we don't need it to scale to 100 million users
24:22
because most of the products, most of the products that we are
24:26
going to build, we when I say we, I mean like any PM, any
24:29
company, any product that we are going to build is going to have
24:33
a very narrow use case. We don't need the internal
24:36
model, the model behind it, to be able to answer log questions
24:40
and medicine questions. And something about LinkedIn and
24:44
something about Google. So go to market, test this
24:47
quickly, see what works, see what doesn't work and then try
24:51
to replace it with a narrower model or smaller model which I
24:55
believe will in a lot of cases perform better than whatever LLM
25:00
model you are using because now it's pre trained on something
25:03
very specific, a 100%. It's funny because it's like you
25:08
almost do full circle, you see if something works and I could
25:12
see that you get the velocity really out the gates.
25:15
You're like does it feature with AI work?
25:18
Yes. No.
25:19
Boom. We can test it.
25:21
OK, we're seeing some good signals.
25:23
Now let's bring it back. Let's start to figure out how we
25:26
can use this in house. I mean, I do know there are
25:30
other people that have come on here and said, yeah, you know
25:33
what, actually we don't even care that we are spending this
25:38
money with Open AI because it is making our product more
25:43
attractive and then people will sign up for our product more.
25:47
So the ROI is there. And so figuring out that ROI is
25:50
a little bit tricky. Some people have it very clear,
25:52
others don't. Yeah, absolutely.
25:55
And actually this is again like 6 or 10 months ago, I was doing
25:59
this as an exercise for something that I wanted to build
26:02
and even generally speaking. So I was thinking, OK, if I want
26:06
to build a Gen AI solution and if I want to test it with 50,000
26:11
people and if I do it for 30 days, hypothetically speaking,
26:15
and I do. So I did all of my, I put all of
26:17
my assumptions in a table, what kind of QPS would I be looking
26:20
for? And then when I use the open air
26:23
public pricing? To calculate how much it would
26:26
cost to run this experience. I was like, wow, this is really
26:31
cheap. No, because I was thinking about
26:33
it from obvious DNA price scale. But test something with 50,000
26:38
users. And just do this back of the
26:43
envelope math. This is really good.
26:45
Otherwise, if I have to build my own e-mail solution, there's GPU
26:49
cost, there's engineering time, there's so much infra that one
26:52
would have to build and then maybe it works, maybe it doesn't
26:57
work exactly, and then we have to do something all over again.
27:01
Yeah. So then when you were talking
27:03
about open source, what is your thoughts on that and how do you
27:06
see that playing into it? With open source, there's a lot
27:11
of I I think Open Air released, it's like new policies where
27:14
they are saying if you're an enterprise customer, we won't
27:17
use your, I am not sure, so don't quote me on that, but
27:20
maybe there's data protection over there.
27:23
But maybe, maybe, maybe, yeah, we'll find out.
27:27
But anyway, open source models are just like open by
27:31
definition, so there is no transparency issue over there.
27:34
People can host the whole model in house.
27:36
So for use cases that have stronger requirements, maybe if
27:41
it's related to investment banking or if it's maybe related
27:45
to law or something else, now you can know and explain what's
27:50
going on. Well, you can completely
27:52
explain, but you can. Controlled what sources are
27:56
going into your model. So that's always good.
27:58
And I think these models open source models are much smaller
28:01
than a lot of other models that are out there, which makes it
28:04
even more cheaper. You can.
28:06
Continue using this large language model for your use case
28:10
without it getting too expensive after, I don't know, 30 days.
28:14
Totally Totally. Yeah.
28:16
Yeah. So you spoke about you playing
28:20
around with the spreadsheet and trying to build out something on
28:22
your own. When you are building a
28:25
generative AI experience, what are some things that you're
28:29
focusing on? Because maybe there's stuff
28:32
around model selection. You're prompt structure, your
28:35
prompt evaluations iterating on prompts, an so with your
28:40
project, how did you look at all that and how did you approach it
28:44
from your product frame of mind? Yeah, when you ask that
28:49
question, I I saw so many flashbacks up.
28:51
Walking through models, late nights, weekends, all that good
28:57
stuff. I don't know.
28:58
Lot of interesting things. A lot of interesting things.
29:01
But anyway, so again, without getting into too many technical
29:05
details and like what the engineering decisions were and
29:08
whatnot, one thing like I mentioned, my general stance is
29:12
see what can get you a good go to market speed.
29:16
Start with that. It's not going to be even if
29:19
someone starting their new product.
29:22
In in. Actually, in that environment,
29:24
speed matters even more than other things.
29:27
So look at what's the easiest to use and start with that.
29:31
So that's from the model perspective.
29:33
Again, it obviously has to solve for whatever your use case is.
29:38
So that's that. The second bug is the prompt
29:42
prompt selection. The one thing that I've been
29:45
like preaching in pure circus, like whoever.
29:48
I speak with other colleagues, colleague, not not, not just
29:51
colleagues, like people in the industry, like, hey, the first
29:55
time you hit a prompt about a product that you want to build,
29:57
you'll see the output and you're like, Oh my God, this is
30:01
brilliant. I've had jackpot.
30:03
But when you do this for 10,000 different experiences, so the
30:09
exact same experience, but for 10,000 different people, 10,000
30:12
different users, you'll see all kinds of anomalies.
30:15
And that is where prompt engineering and from iteration
30:19
has to come in. And then you have to actively
30:21
squash these bad experiences and make sure that you're building
30:25
as consistent and experience as you can while it's still
30:29
obviously providing great value to the end user.
30:32
Mm-hmm. And is that the consensus?
30:33
Is it mainly the prompt that you're focusing on more, as
30:37
opposed to catching it if it outputs something wrong?
30:44
When catching it when it output something wrong, then you would
30:47
need an external. Evaluator and then yeah, in that
30:51
sense. So when you're trying to build
30:53
real time experiences, we can't do an external evaluation at
30:57
that point or maybe like a near line evaluation.
30:59
So it has to be built into the system.
31:02
I think the first line of defence is still to have a
31:05
better prompt rather than you can always make two GPA calls
31:10
and have the second one verify the first one.
31:13
It's going to help and it is one of the valid techniques that
31:16
people use out there. But again, now your experience
31:18
calls double what it was. Going to cost Collier, so
31:22
there's a lot of value in prompt engineering.
31:25
There's a lot of value in prompt iteration, so one should first
31:29
spend time in making your prompt extremely robust.
31:33
See what experience, how much of the mountain can you climb with
31:37
just this one problem? And then when you start hitting
31:41
the limits of that prompt, then you can think about, OK, what do
31:44
I still need? What?
31:46
How do I make that happen? Yeah, yeah, yeah.
31:51
The idea of what you were saying when you were when you talk and
31:55
you preach about the prompts and just getting anomalies and how
31:59
basically every time you input the same thing the output can be
32:05
a little bit different. Like how are some other ways?
32:09
Is it just the prompts that you're looking at to help build
32:14
that in a better way or are there other things that you're
32:18
doing? Like I know we've had people on
32:19
here from different tooling companies or who have built open
32:23
source tools. Like I'm thinking about this guy
32:25
Maxine Buchem Scheme, who is he created Prompt Demise.
32:31
And I know there's a tonne of tools in that space where it's
32:34
just like helping you evaluate your Proms, helping you save
32:37
your Proms, helping you do everything that you could ever
32:41
want and imagine with your prompts to make sure that you're
32:46
tweaking them and you're getting the most you can out of the
32:49
prompts. And it feels like that's great.
32:52
And then there's also a guy that we had on here just a a week or
32:57
two ago who was CTO of Beto and he was talking about how you
33:01
want to just basically make the constraints as small as
33:08
possible. So you're really trying to
33:11
constrain the model and make sure that you give it a tonne of
33:16
context, and then you constrain the output in the prompt.
33:20
Did you come up with any other tricks or do you have any other
33:23
secrets to lead us on to? Yeah, there's I I I very much
33:29
agree with the second person who mentioned like add as much
33:32
context as possible and then constrain the output for
33:36
whatever you want to get out of the model.
33:39
So that's one thing. What other things that we look
33:42
at, obviously temperature is 1 and then all of the practise has
33:46
no OK. If you want something much more
33:48
consistent you can look at and you can look at the temperature
33:50
parameter. But again, that comes with its
33:53
own pros and cons. Anyway, that's that.
33:56
I love the part that you mentioned about tooling, because
34:00
as we keep doing this, more and more tooling gets extremely
34:04
important and that becomes a productivity bottleneck if you
34:08
don't have good tooling. So that's something that's going
34:11
to be very useful. But I'm not recommending
34:14
anything specific right now because again, these things
34:17
change every month or every two months.
34:21
When I started, which was many, many months ago, there were no
34:25
tools around. So we had to build our own thing
34:28
and see how we can make it work. That's that.
34:30
Now I think Microsoft has Microsoft guidance and there are
34:34
so many great open source tools out there.
34:36
And last week I think, or the week before that, I read a paper
34:40
where they're like AI can create better.
34:43
Drums air has like proven to create better prompts than
34:46
humans. For so maybe we automate this
34:50
spot out as well. To some extent we still have to
34:53
be involved, but to some extent we automate that.
34:56
Yeah, yeah. And really reminds me of this
35:00
conversation that I had with in The name is failing, you know,
35:04
but some Stanford researcher. And he was, I think his name was
35:08
Chiron and he was talking about how his whole pH.
35:13
D was on where to have a human in the loop.
35:17
So if we're going to be having a human in the loop in
35:20
increasingly less places, he wanted to make sure that we
35:25
choose the right places to have them in the loop.
35:28
And so it feels like that is just more and more relevant.
35:31
Yeah, new product comes out or each new update comes out.
35:37
I love that. I love that, actually.
35:38
Like after the podcast, send me a link to whatever issue such as
35:42
I'd love to read it will do for sure.
35:45
So when you were building your tool we were talking about
35:49
earlier how don't just go out and set out to build with Gen AI
35:54
just because. But did you do that?
35:57
Did you like not follow your own advice and do as I say not as I
36:00
do. Type thing and go and try and
36:03
build something just to get your hands on Gen AI?
36:05
Or was it that you saw a problem and you said alright, I could
36:09
solve it with XYZ, but I think the best and the fastest is
36:13
going to be with AI. So there there are a couple more
36:19
extra data points over here. So I've been in Word and Jenny I
36:22
for a few years now. So fun fact, 2017 I was working
36:28
on some research item and I was using Gans to to generate
36:33
things. The gang what?
36:34
I know, right? It's old now.
36:36
Yeah, I went out of style quick. Yeah, yeah.
36:40
So I was inputting text, then out putting an image out of it,
36:43
but then the only image I could show it was a 28 by 28
36:46
thumbnail, which was really small, obviously takes a lot of
36:48
CPU capacity to do anything. So from that world, I knew about
36:53
Jenny already and I think September, October of last year,
36:57
I was like, hey, there are a lot of new papers coming out and it
36:59
seems like there's. Better solutions now and the
37:03
technology is potentially commercially ready.
37:07
So that I had that at the back of my mind and at LinkedIn I saw
37:12
creators. Create this among other things.
37:15
Now I'm working on content understanding as well, but
37:17
anyway. In any kind of product, there
37:20
are going to be some problems that were very difficult to
37:23
solve. And then like, hey, if I add Jen
37:26
AI, I can actually tackle this problem now.
37:30
Until now I probably couldn't have, but now Jenni has or
37:35
charged app at the time has made it accessible.
37:39
For me to tackle this problem, and that's how I was looking at
37:42
it. So it was a bit of an unlock
37:46
where you realised, OK, now it's finally time to concoct my world
37:52
domination plan that I've been waiting for.
37:56
But no, in all seriousness, we mentioned we were going to talk
37:59
about what you're doing at LinkedIn.
38:01
And I don't know if you know this, but I'm a bit of a creator
38:04
on LinkedIn myself. Yes, absolutely.
38:07
Always focused on my feed when you bought something.
38:11
So basically you work for me and yes Sir, yes Sir.
38:15
What is? What are some cool things that
38:17
you're doing at LinkedIn? Are and are you the person that
38:21
created the features? So that I can just put a prompt
38:25
into my post and then it will create some kind of a I
38:29
generated post? OK.
38:32
So what do I do? I think then I started a couple
38:35
years ago. So back then it was more about
38:37
personalization and what I can do to sort of creators on the
38:42
platform. Now as time as it well it well
38:46
now I'm focusing on how can I use AI to serve creators on the
38:50
platform. This can mean better
38:52
distribution for them. This can mean getting smaller
38:55
creators to grow, reach that audience thought that virtuous
38:59
flywheel of engagement and make connections.
39:02
So that's something that we actively work on.
39:04
I'm also now working on content understanding, so seeing how we
39:09
can interpret what a particular post is about, what a particular
39:13
image video is about. And then you start to serve
39:17
better matching content to you when you log in on LinkedIn.
39:20
And once you create something with Elms introduction, there's
39:25
someone else out there who is interested in alms and get your
39:28
content to them. So that's something that I
39:31
actually work on. Sounds amazing.
39:33
I want all of that get get my stuff out there so I can start
39:37
sponsoring my posts and make a living from it.
39:41
Become a real. Time on it.
39:43
I have a meeting in 10 minutes with my engine manager, so I'll
39:46
just ask her to just make everything happen today.
39:50
The thing that I also was interested in on like you as a
39:56
product guy, and maybe this isn't necessarily a LinkedIn,
40:00
but just like what you've seen being able to go from what you
40:04
were talking about before, where back in the day you had to put a
40:08
lot of work into again and you had to use a lot of GPU
40:11
resources just to get something that kind of worked right?
40:15
And now it feels like over a weekend you could probably hack
40:18
together an MVP and show value for something.
40:22
So it almost has this gigantic. Unlock for any product engineer
40:28
out there, because now there's no excuse as to why something
40:33
isn't being created, or at least an MVP isn't being tested.
40:36
Have you found that to be true, or am I just like making that up
40:40
and creating my own theory and thesis?
40:43
And my head no, it is 100% true, 100% true analogues, so many
40:48
opportunities, so much speed. I think this is one of the best
40:52
times to be an APM because generally as PMS. You want to
40:56
test your ideas or do you want to test your hypothesis?
40:58
See what works, what doesn't with a solutions?
41:00
It used to be extremely tough and difficult and now you have
41:05
this option where you can potentially even test for a
41:08
small period with off the shelf solutions and then later on you
41:13
can choose to build your own. Yeah, exactly.
41:16
It's almost like you get this opportunity and you can I say
41:22
ohh, actually if we wanted to do this, I could probably make that
41:26
happen and it's a few API calls. So let's just test it out.
41:30
Let's see if you're getting those signals and how are you
41:33
looking at signals in general. Like I know each use case is
41:39
going to have its own distinct signals and distinct things, but
41:46
when you're trying to figure out like how to measure ROI on your
41:51
different features, how do you think about that and like, walk
41:54
me through your the way that you go about it?
41:58
Yes. So the 1st I'll start with the
42:02
channel playbook again a rough idea of a playbook that I
42:06
wouldn't have anyone go through. Its first decide what problem
42:10
are you solving. Is it a problem?
42:13
What solving do your users actually care about you solving
42:17
that problem? So that's the primary, that's
42:20
where callback to one of our earlier points where I'm like,
42:24
hey, Chennai is just a tool to solve problems for users.
42:29
So see what kind of following that you're trying to solve.
42:32
Start from there and then you can set a time period.
42:35
You can set how good your MP is supposed to be, and you can test
42:39
it against what kind of metric you're trying to move versus.
42:42
I wanted how many dollars it's going to cost you to build this
42:44
experience and then you can see in the long term Test match
42:48
workout. But the fun part is, when you're
42:51
thinking about general experiences, it's not all of
42:54
this very obvious, very normal playbook things.
42:58
You also have to think about, OK, now this is a new
43:01
experience. This is not even just a new
43:04
experience, it's a new interaction pattern.
43:07
It's a new paradigm completely. So what are new things that I
43:10
have to think about? And over there, I think trust is
43:14
a very big part that we have to now actively think about and
43:17
everyone should actively think about this.
43:20
Because any of these LM models, they're very confident they will
43:25
say something, but it doesn't always make sense.
43:27
And when you want to survey users, you want to be right.
43:31
So. Building of feedback mechanism
43:34
and actively tracking that. Seeing.
43:37
First of all, do a lot of offline testing.
43:39
Obviously I'm assuming that's done already, but when the
43:43
product is online, have a feedback pipeline.
43:45
This shows that to your users that you're willing to learn.
43:49
You're trying to be fast. You're trying to push something,
43:51
You're trying to create new experiences for them.
43:54
But you're also willing to take feedback and you will see corner
43:58
cases where things just blow up and you would have never thought
44:01
of that corner case. And then keep track of what's
44:05
not working and actively fix that.
44:07
Because with a lot of these new experiences it's you are
44:11
creating a voice for your brand also has voice, so that's a
44:15
different thing. But when Jenny experiences,
44:17
you're adding this human element.
44:20
So you have to make sure that the bar is very high for trust,
44:23
because if it says something stupid, it's going to reflect
44:27
badly on your product, on your company.
44:30
So yeah, trust is the new element that I would supercharge
44:33
for genuine experiences. And so you as a PM coming into
44:41
the, I guess you've been doing this for a while.
44:43
So potentially, it's not exactly your specific case, but I'm
44:49
wondering if you've seen the other people go the other
44:55
direction. Whereas you're doing a lot of PM
44:58
work and you're understanding the technical pieces and you're
45:02
going deeper into the machine learning and the alms.
45:05
And what I've seen from different PMS is there's been
45:08
this huge unlock with LM's, right?
45:11
And so now they can hit an API and get this whole new world
45:18
opened up to them. And they're like, wow, this is
45:21
incredible. All that stuff, like you saying,
45:23
like that stuff that I wanted to build for the last three years,
45:25
I can do it now. This is great.
45:28
And so that is one side where it's like PMS moving more
45:32
towards understanding. You're trying to understand AI.
45:36
Have you seen it where the mill teams or people that are doing
45:40
mills, data scientists, whoever are trying to move towards
45:44
product a bit more, It almost feels like potentially because
45:49
now all of these PMS are basically playing in their
45:52
playground. They're like, alright, whoa,
45:54
whoa, product guy, what's going on here?
45:57
Like, I gotta. I guess I gotta understand your
45:59
world a little bit more too. Yeah, yeah.
46:03
This is one thing that I've actively noticed and I really
46:07
respect in the people that I am working with.
46:10
The only very product oriented. And this has been true long
46:14
before Jenny. I came in because for a
46:18
different experience for a non machine learning experience.
46:21
Sometimes it could be as cut and dry as hay.
46:25
Product and design worked on some design and we need this
46:29
exact experience and let's let's build it.
46:33
But with machine learning, it's always going to be, hey, this is
46:35
the experience that I'm trying to generate.
46:37
And then the machine learning engineer is the one who was
46:40
actually quoting the model. So they know where the biases
46:44
could come up, They know how the experience could go wrong, and
46:48
it's very difficult for someone who is just a PM to know all of
46:53
this or actively squash any of those.
46:56
Which was before they come UO. The people that I've worked
46:59
with, they are very product oriented.
47:00
They actually think about what's the problem that we're trying to
47:03
solve, what could be the corner cases, where are we potentially
47:07
adding bias, how could we actively debias the system?
47:11
So I think all a lot of successful machine learning
47:15
engines that I've worked with. They're very product oriented
47:18
and as a PM I'm very thankful for it.
47:21
Yeah, I bet I want to ask you about the different gotchas that
47:25
you found. Maybe you don't have to talk to
47:27
us about this stuff at LinkedIn, Wink, Wink, but you can talk to
47:33
us about things that you were saying how it feels good when
47:37
you add a new feature or you bring this AI into your product
47:42
until all of a sudden you start talking to customers and they're
47:47
saying this is absolute crap. And you mentioned how that trust
47:51
factor is and needs to be very, very boosted.
47:57
And So what are some other gotchas that you've seen the?
48:01
God shows are also just things that I've been actively trying
48:04
to be very particular about is. Thinking about what are you
48:10
building the experience for? So the evaluation criteria, what
48:14
makes this experience an actual success, And that's something
48:17
that you will have to define like way earlier in the cycle
48:20
and you have to be. Fanny strip in what the
48:24
constraints are. What are we actually trying to
48:26
build? Because when you're going to
48:27
give a prompt or any input to any of the LLM models, they can
48:31
create an output. Now how do you save this as good
48:34
or bad? It's very difficult to sit and
48:37
evaluate and be like, OK, maybe I would give this A5110, but
48:41
what is A5110? And then when you want to scale
48:44
the experience, you will have many people trying to score it.
48:48
So OK, now this person's favourite.
48:50
That person's 10. And so having that evaluation
48:54
criteria is extremely critical so that you actually know what
48:57
you're testing for and then you can reverse engineer your prompt
49:00
to actually solve for that criteria.
49:04
So that is something that I would say people should actively
49:07
think about and then having a cadence going on, prompt
49:11
engineering, saying what kind of problems you can solve, what
49:13
came up in the previous evaluation, how can you do it
49:16
better this time. That's that.
49:17
And the third piece is then trust.
49:20
See, once you look at what kind of feedback you're getting, how
49:25
what kind of patterns are now emerging, the models change.
49:29
A lot of things change. People's interaction patterns
49:31
change. Because a lot of times you're
49:33
going to build an experience that people won't use to in the
49:37
old world. So you have to check your bias.
49:40
This is actually a big one. You have to check your biases on
49:43
how you thought the users will use the ecosystem or the new
49:46
feature. Yeah, and maybe they have a
49:49
completely different style of using your.
49:52
System and then charge. GPD could give a third response
49:56
that you hadn't even thought about.
49:58
So that's something that comes right out of left field.
50:02
Yeah, yeah, that's where you knew it.
50:04
And I love this idea of like, the evaluation, man.
50:07
It's so hard. And This is why we put together
50:09
that survey and it feels like the consensus is just throwing a
50:13
thumbs up or thumbs down on the response and letting the end
50:18
user say if it was good or bad. But man, like, the truth is
50:22
that's not good enough and that's not going to get you very
50:25
far. And especially if you're trying
50:27
to figure things out. Like half the time when I get
50:30
bad responses, I don't give a thumbs down.
50:33
I just am like, ah, this sucks. I gotta go figure out another
50:36
way to do this. And so I I know we've had a loan
50:40
on here from Aporia and he was talking about how he looks at
50:44
when he's monitoring the output. The thumbs up, thumbs down is
50:49
pretty much like, it's not worthless, but it's next to
50:52
worthless. And he looks at things like if
50:55
you bounce after you get the output, it's like, oh, that's a
50:59
pretty clear ignore that that output was not useful for you.
51:02
But then there's like those false positives where it's like,
51:05
oh, you just copy paste it and you bounced.
51:07
And so it's super complicated to figure out like was this
51:10
valuable or not. I don't really know how to
51:14
figure that out and have a clear picture on it.
51:17
And again, that's why we put out that evaluation survey and
51:20
hopefully through the collective wisdom of all of us in the
51:25
community, we will figure something out or we'll have some
51:29
clear ways of doing it and some of these best practises and
51:32
patterns will emerge. But poop right now, it's a
51:35
jungle out there, man. Yeah, yeah it it's a very
51:38
exciting jungle. So I I've before you mention the
51:41
bounce, I was going to mention the bouncer as well and I have
51:44
my own thoughts are like what, what signals one could
51:46
potentially look for, but again that would kind of vary from one
51:51
product to another as well. Yeah.
51:54
So dude, last question for you before we finish this one off.
52:00
Who is more confident, A 14 year old you Bollywood actor or
52:07
ChatGPT? Oh oh.
52:11
I think I was definitely more humble than child CBD.
52:16
If you go outside, typically they wake ohh yes I am an actor.
52:20
Like no you're just an I and then on the output like I am an
52:24
AI machine learning model. But anyway, yeah, it's it was
52:27
the most probably 1. Yeah, yeah, I was laughing
52:32
because I I made the joke at the beginning of the year and I said
52:36
my New Year's resolution is to be as confident as Chad GT and
52:41
that. Yeah, and the years, not over
52:43
yet, but you know, still working on it.
52:45
No, it's still trying to get there.
52:47
They don't think I can. It's a pretty high bar that
52:50
they've set. So.
52:51
I love that. I love that for you.
52:55
Well, this has been great. I really appreciate you coming
52:57
on here and talking to us about building products, especially
52:59
with the generative AI. And so I'll let you go.
53:04
That's it. We'll end it here.
53:05
Go. Awesome.
53:06
Thank you so much for having me. There are serious.