Latest / The Pragmatic Engineer / AI Engineering with Chip Huyen

Transcript

AI Engineering with Chip Huyen · The Pragmatic Engineer

0:00
SPEAKER_0: How would you define AI engineer or AI engineering?
0:02
Yeah,
0:03
so before when you wanted to build machine applications,
0:06
you need to build your own models.
0:08
So that means that you need your own data and you need expertise in how to train
0:12
a babysitter model.
0:13
However,
0:13
nowadays if you want to build your application leveraging machine or AI,
0:16
you can just send a direct API call and access is this wonderful capability.
0:21
SPEAKER_1: So that's really, really lowers the entry barrier to people.
0:24
SPEAKER_1: You don't need data anymore, you don't need a fancy AI degree anymore.
0:28
It's a shift of focus from less machine learning and more engineering and
0:33
more product.
0:34
Chip Huyen is a computer scientist and writer and author of the book AI Engineering.
0:39
SPEAKER_0: This book is currently the most-read title on the O'Reilly platform.
0:42
Previously,
0:42
Chip was a researcher at Netflix,
0:44
a core developer of NeMo,
0:46
NVIDIA's gen AI framework,
0:47
ML Engineering's Snorkel AI,
0:49
and founded and sold an AI startup called Claybutt AI.
0:53
She taught machine learning systems design at Stanford and her current book
0:57
is a second one on ML and AI Engineering.
0:59
It's safe to say she's one of the most-read ML engineering and AI engineering
1:03
experts in the world.
1:04
In our conversation today,
1:05
we cover what is AI engineering and why does this feel a lot more full-stacked
1:09
than ML engineering did.
1:11
What are typical steps to build on AI application from choosing a model
1:15
through using RAG all the way to fine-tuning?
1:17
What are practical ways for software engineers to get started building AI
1:20
applications and a lot more on this very timely topic.
1:24
If you enjoyed the show,
1:25
please subscribe to the podcast on any platform and on YouTube.
1:28
SPEAKER_0: Thank you.
1:29
SPEAKER_0: This greatly helps the show get to even more listeners and viewers.
1:32
SPEAKER_0: Chip, welcome to the podcast.
1:34
SPEAKER_1: Hey, hi, I'm Chip.
1:35
SPEAKER_1: I'm very excited to be here.
1:37
So I've been following your substack for a while,
1:40
so I was really looking forward to the chat.
1:44
SPEAKER_0: So first of all, I really wanted to congratulate you on this book.
1:47
SPEAKER_0: I've started to read the book, so I've not read the whole thing.
1:49
I have started with some chapters and I went deeper in others,
1:52
and what I found is when I looked at a table of contents,
1:55
I was like,
1:55
"Well,
1:56
this looks okay in terms of,
1:58
it'll span a broad sense because it goes from how do you understand
2:03
foundation models,
2:04
how do you evaluate them,
2:06
whatispromptengineering?"
2:08
And you go into things like fine tune RAG,
2:11
fine-tuning dataset engineering,
2:13
but then on each of those sections it just starts to get...
2:16
like,
2:16
initially there's an introduction,
2:18
but it starts to then go deeper.
2:19
So for example,
2:21
for evaluation mythology,
2:22
like here I'm just looking at a table of contents,
2:24
but I start to read this,
2:25
it's like,
2:26
well,
2:26
we know it's important to evaluate AI models.
2:28
We know it's harder to do,
2:30
but then you go into things like AI as a judge or ranking models with
2:33
comparative evaluation and challenges of it.
2:37
And then it goes,
2:37
that's where some parts,
2:40
in my sense is that I had to slow down,
2:42
I had to look things up.
2:44
So it does go really deep into a lot of these sections,
2:47
which I found very refreshing that it's got a mix of breath,
2:52
but it also goes deep.
2:53
So this is definitely not a fast read for me,
2:57
but it's one of those things where I'm just going to keep coming back to it.
3:01
SPEAKER_1: Thank you.
3:01
SPEAKER_1: It was not a fast write either.
3:03
SPEAKER_1: It took quite a while and a lot of references.
3:06
I think I also published,
3:07
so I think in the book I cited about over a thousand references.
3:12
SPEAKER_1: So I actually read even more papers.
3:15
SPEAKER_1: I gone to a lot of code bases.
3:16
I have a tracking of 1000 reposts,
3:20
at least now 800 Stars now,
3:23
on GitHub Stars.
3:23
So I went to a lot of those code bases and a lot of blog posts,
3:28
other books from the '80s,
3:29
from the '70s,
3:30
'90s to understand AI.
3:32
So I also publish at least about approximately 100 links,
3:36
reference links.
3:37
SPEAKER_1: I felt really, really useful for me in the process of writing the book.
3:40
So if you just want to look at those references on their own,
3:42
it's on my GitHub.
3:43
SPEAKER_0: And I was surprised with some of the really original research.
3:50
It's not just like,
3:51
oh,
3:51
here,
3:52
I wrote about these papers or here was what I read.
3:55
But as you mentioned the 1000 reposts,
3:57
you actually have a paragraph or a section about how did GitHub repositories
4:03
change over time,
4:04
the ones that were about infrastructure AI,
4:07
application level,
4:09
other things,
4:10
andyouactuallyhave-
4:11
SPEAKER_1: More development.
4:11
SPEAKER_0: More than 900 reposts mapped out.
4:13
SPEAKER_0: I never saw anything like that.
4:14
And clearly that was you kind of slicing and dicing and doing your own kind of research.
4:19
SPEAKER_1: Yeah, I feel like I have this thing, I do a lot of manual labor.
4:24
I do think I get a lot of value out of doing things in non-optimally,
4:28
I feel like a lot of people focus on what is the quickest way to do it,
4:30
what is the funniest way to do it?
4:32
But sometime if you are willing to put into effort into things that a lot of
4:36
people are not willing to,
4:37
I feel like you can get some kind insight that other people don't get.
4:43
So one thing that is a little interesting about this book,
4:44
it is about AI engineering and this field moves so quickly.
4:47
Like,
4:47
just in a week we've now had a new model come out,
4:49
for example,
4:49
DeepSeek,
4:50
that people are talking about in a few weeks.
4:52
SPEAKER_0: How did you write this book?
4:54
How were you able to write a book about such a fast moving industry so that by
4:57
the time it's released,
4:57
which was clearly a few months after you finish it,
4:59
it will still be relevant?
5:06
SPEAKER_1: That is a great question.
5:07
SPEAKER_1: So when I started writing the book, I was thinking the same.
5:09
I was like,
5:09
"Is now the right time to write it,
5:10
becausethere'ssomanythingsarestillchanging?"
5:12
But when ChatGPT really came out,
5:13
a lot of people,
5:14
I had this existential crisis.
5:15
I was in a group chat and I was just like,
5:18
"Oh no,
5:18
whatdoesitmeanforusengineer?"
5:20
SPEAKER_1: I feel like there are two things that I usually identify with.
5:22
SPEAKER_1: One is being an engineer and the other is being a writer.
5:24
SPEAKER_1: And guess what?
5:24
Two use cases that AI are really good at,
5:26
writing code and writing,
5:27
right?
5:37
SPEAKER_0: Yeah.
5:37
SPEAKER_1: So it was like, oh shoot, well what does it mean?
5:38
SPEAKER_1: So I started interview a lot of people.
5:39
I start like reading so much and I talk to a ton of people and I started making a
5:42
lot of note and as a process what I realized is that while a lot of those things
5:47
seem new,
5:47
a lot of the fundamentals have been there for a while.
5:50
SPEAKER_1: So for example, language modeling is not a new task.
5:52
Claude Shannon introduced that back in the 1950s as a time level,
5:56
we'll talk about RAG,
5:57
right?
5:58
SPEAKER_1: RAG is actually not new, is based on retrieval augmented generation.
6:15
SPEAKER_0: RAG, as in retrieval augmented generation, right?
6:17
SPEAKER_1: And retrieval is a very old technology.
6:18
It is powering,
6:20
it's already powering a lot of use cases on the internet,
6:24
like search or recommender systems and better databases have been around
6:27
for a while.
6:28
SPEAKER_1: In fact, the search is very, have so many cool algorithms already.
6:31
So I thought,
6:31
"Okay,
6:31
first I see a lot of things that are new," and the second is that I try to focus on
6:36
like asking the question.
6:37
Okay,
6:37
so sometime it was like,
6:37
oh there is a problem with this,
6:39
a solution to this problem.
6:41
And I asked the question of whether this is due to fundamental limitations of
6:44
AI or is this just due to the temporary capabilities of AI.
6:46
And I try to see,
6:47
okay,
6:47
if it's due to something of the recent current capabilities,
6:51
how fast is that capability changing?
7:07
So first of all,
7:08
in the early days a lot of people have shared a lot of prompt tips,
7:11
like for example you try to prize the models,
7:13
like,
7:13
"Hey,
7:13
say,
7:14
if you answer this correctly,
7:15
I'mgoingtogiveyou$200."
7:16
Or we talk about prompt robustness,
7:17
how robust is a model to prompt perturbations.
7:20
And then I was reading about it and I found is actually models is getting more
7:22
and more robust to prompts.
7:23
First of all we saw from the GPT,
7:25
to GPT 3.5 compared to GPT-3,
7:26
it's already so much more robust.
7:28
That means that the small changes to prompt actually reduce to a lot less
7:31
variations in model performance.
7:34
SPEAKER_1: So this type thing, I felt like it's probably not going to stick around.
7:38
SPEAKER_1: So those tips are not going to be very relevant to share.
7:59
So already as people were still like,
8:00
this was at the height of people are saying there might be a job as a prompt engineer.
8:04
SPEAKER_0: So you already saw this is likely trending down.
8:05
SPEAKER_0: Do I understand it correctly?
8:07
So yeah,
8:07
I do think just try...
8:07
so writing I think it's a score is making a bet.
8:10
Like when you write a topic,
8:11
you're trying to bet on whether it's going to stay relevant in the future.
8:15
So I think it's the people have been trying to looking at progress and trying
8:20
to see what it's going to be like in one or two years from now on.
8:29
SPEAKER_1: So first another example is context length.
8:31
So at that point it was like okay,
8:33
we want long context length,
8:35
but then I kept seeing these people going really,
8:37
really fast from what,
8:31
8,000 context length to 128 in a few months.
8:39
SPEAKER_1: It's super, super fast.
8:39
So it was like okay,
8:40
maybe the question is less about context length but context efficiency
8:47
because it's like can a model use a context efficiency really,
8:52
really well.
8:53
So there are certain,
8:53
so those are the best and I do think that there are certain changes during the
8:59
process of writing the books that make me feel more confident in the best I made,
9:02
or like another things like multi-modality,
9:06
I do think this.
9:07
When I wrote about multi-modal back in 2023,
9:09
people told me,
9:10
"You're too early.
9:10
SPEAKER_1: Everyone was still working on language now.
9:12
SPEAKER_1: We are not there yet."
9:13
SPEAKER_1: But it was like, yeah, it's just inevitable.
9:15
I think we learn how to work with language,
9:17
but I do want to do a lot of stuff with more than just language and I think it's
9:23
just like,
9:23
and nowadays just everywhere,
9:24
almost all models nowadays are like multi-modal.
9:38
So the title of the book is AI Engineering and we also now have this term of AI engineer,
9:43
it's spreading like wildfire.
9:45
How would you define AI engineer or AI engineering because I feel it's a
9:48
little bit of a loaded term these days?
9:54
SPEAKER_1: It is.
9:54
SPEAKER_1: I feel like a lot of terms nowadays are loaded.
9:57
You're not allowed to use agent anymore,
10:00
you're not allowed to use a lot of things anymore.
10:04
So when I was agonizing over the title for the book because people was like...
10:08
first I know that we need a different term for machining engineering,
10:12
and the reason is that with foundation models there are a lot of fundamentals
10:17
like systematic approaches that still the same at machining engineering,
10:21
there are a lot of new things.
10:22
So for example,
10:24
one thing is that before when you wanted to build a machining applications,
10:28
you had to build the old models.
10:36
SPEAKER_0: And so just be clear, you're comparing it with machine learning, right?
10:36
SPEAKER_1: Yes.
10:38
So what machine learning engineers did versus what AI engineers are doing.
10:43
SPEAKER_1: Yeah.
10:43
So yeah,
10:43
just trying to explain why we need a different term for machining engineers
10:47
to describe what we are doing today.
10:49
So before when you wanted to build a machining application,
10:53
you need to build your own models.
10:55
So that means that you need your own data and you need expertise in how to train
11:00
a babysitter model.
11:01
However,
11:01
nowadays if you want to build an application leveraging machine link or AI,
11:06
you can just send it direct API call and access this wonderful growability.
11:11
SPEAKER_1: So that's really, really lowers the entry barrier to people.
11:14
SPEAKER_1: You don't need data anymore, you don't need a fancy AI degree anymore.
11:16
A second thing is that before you need distributions because you deploy
11:20
machining application as part of the CD applications.
11:23
So first of all you build a recommender system,
11:27
you need an e-commerce website so that you can have recommender,
11:30
or some kind of website.
11:38
So it's deployed as part CD application,
11:41
fire detection is part of making banking applications or some kind of
11:44
payment app.
11:44
SPEAKER_1: But however, now you can just put it out as standalone applications.
11:48
You don't need an existing distribution channel,
11:50
but having a distribution channel is really,
11:52
really useful.
11:59
So I think that another very big thing is that it's a shift of focus from less
12:02
machine learning and more engineering and more product.
12:05
So before you start from,
12:05
if you good machine engineer,
12:07
you start from data,
12:08
you have to gather data,
12:09
you maybe had human annotations and then you train a model and now once a model
12:14
is good,
12:14
deploy that into the audio product.
12:16
But nowadays you actually start with a demo so you have a good idea,
12:19
okay,
12:19
let's just try it out and see if it works.
12:22
So we start a product and after you say,
12:26
"Okay,
12:26
sounds like it works pretty well,
12:28
Iwanttomakeitbetter."
12:37
So they started gathering more data maybe a bit as more of problem examples or
12:40
in very rare case I don't recommend most people do in the early days it's like fine-tuning,
12:45
but it's very rare.
12:47
But basically you start getting more data maybe for evaluation,
12:52
extremely important.
12:53
Having good evaluations become even way way more important with AI Engineering.
12:57
So you've got data and then after that it was maybe you've been sending a lot of
13:00
API calls to OpenAI and Anthropic or Google and okay,
13:03
now it's too expensive,
13:03
"NowIneedtodevelopmyownmodel."
13:10
So you start hosting your own model using open source alternative or
13:13
fine-tuning model.
13:14
Yeah,
13:15
so before machine engineer you go from a data model to product and now with
13:20
engineering you go from product to data into model.
13:23
So it plays a lot more focus on product and data,
13:26
which wouldn't be a competitive advantage when everyone shares a similar
13:30
base like AI capabilities.
13:31
So I do think that says need a different term,
13:34
just separate it from machine engineer and I didn't know what terms to use.
13:36
SPEAKER_1: And then I was like, "Okay, that's just ask the people."
13:39
So I surveyed a bunch of people,
13:40
if I was doing,
13:41
what does this building applications on top 5-digit models and almost
13:45
everyone was like,
13:45
"AI Engineering" and I was like,
13:47
"Okay,
13:47
if that's the case and this,
13:48
let'sgowithAIEngineering."
13:57
So do I understand correctly that the biggest difference is that machine
14:02
learning engineers did a lot more kind of groundwork getting the data and
14:07
building the model.
14:08
Whereas AI Engineering,
14:09
you have a lot of that at least initially you can start either as APIs or something.
14:12
So there's more of an engineering,
14:14
you kind of hack things together,
14:16
you put it together,
14:16
and then over time as things become more serious or your product is bigger,
14:20
you do a lot more of the...
14:21
you might build your own model or you might host it.
14:23
You might one day even build your own model if it is there,
14:26
but it is just a lot later.
14:27
So a lot of ML engineering comes down the path if it's big enough,
14:30
if it works,
14:31
et cetera.
14:31
Whereas with ML engineering it was the other way you had to put in all this
14:36
effort and then see if it even works?
14:43
So first,
14:43
I feel like every,
14:44
companies may have different definition of the role.
14:47
I think it's even the same company,
14:48
people with the same title,
14:49
same role,
14:49
can do very different things.
14:51
SPEAKER_1: So it's not very, never like a clear-cut definition.
14:58
Second things is that,
14:59
I don't think the question is like machine or engineering.
15:02
In the vast majority of general AI system have since there are very strong
15:07
traditional machine learning classifier component.
15:10
So imagine you're building a customer support chatbot,
15:13
which is whenever I ask at a conference,
15:15
I see a lot people raise your hand,
15:17
very classic general applications.
15:18
And so you get a request from a customer and maybe you have several different
15:22
potential solution for it.
15:24
Maybe if it's an easy query you might send it to a cheap model,
15:27
or if it's a harder query,
15:28
you might send it to a more expensive model.
15:29
But if it's something very sensitive like,
15:31
"Hey,
15:31
why did you charge me twice for the bill last month," right?
15:34
SPEAKER_1: Then you might want to send it to human operator.
15:36
So you might have this one like a router or an intent classifier,
15:38
you choose what to send it to,
15:39
which is like a traditional classical machine learning model that you can build.
16:00
Or after you get a response from an AI model,
16:01
you might think it's like okay,
16:02
this is contain like PII because you don't want to send back to users
16:04
responses contain private information.
16:05
SPEAKER_1: So that PII detection can be, or toxicity detections can be a classifier.
16:08
Or in the RAG system nowadays we talk about how a lot of it using retrieval systems,
16:11
which are also I think it's a RAM of classifier,
16:12
classical machine learning that you can build yourself.
16:32
So what are the most common techniques used when building AI applications,
16:37
things that the software engineer who's going into building AI
16:40
applications I should know about and later I can go deeper into?
16:47
So the assumption here is that you have tried a lot of solutions and now you try
16:52
generative AI and you think this generative AI is a solution for you and it think,
16:56
what is the first,
16:57
like what should I progress from there,
16:59
right?
16:59
SPEAKER_1: Is that correct?
17:00
SPEAKER_0: Yeah, yeah.
17:00
SPEAKER_0: It's just like what are some common approaches?
17:03
I think things like RAG,
17:04
fine-tuning,
17:05
or other things that are just good to know about,
17:07
I should probably learn more about those topics.
17:11
SPEAKER_1: Yeah.
17:11
So I think those techniques are useful and I would usually recommend some development,
17:15
not recommend,
17:16
but a common pattern you have seen is a certain developmental power path.
17:20
So initially the first thing it would say is trying to understand what means a
17:24
good response,
17:24
what a good response is and what a bad response is.
17:27
SPEAKER_1: So what you want is a large share rate and it's not always intuitive.
17:38
SPEAKER_1: So for example, LinkedIn has a very great example.
17:39
So they build like job fit assessments for candidates and they found out that
17:43
a majority of their time spending just to understand what candidates needed
17:47
from the model.
17:47
So initially they focus on correct,
17:49
but since they realize it's candidates [inaudible 00:17:58] found it not helpful.
17:53
Say if a candidate asks the model,
17:55
am I a good fit for this job and the AI respond with,
17:57
"You're terrible fit," it was like,
17:58
"Okay,
17:58
wellwhatamIdoingthisinformation?"
18:11
So you need to try to understand,
18:14
okay here is they want more understanding what are the gaps and how they can
18:17
fill the gaps or get suggestion for other roles that are a better fit for them
18:21
right now.
18:22
So have this pictures clear understanding and then build a guideline,
18:26
like,
18:26
okay,
18:26
given this response,
18:28
answer this,
18:29
be helpful,
18:29
show them the gaps or show them the role.
18:31
SPEAKER_1: Have very clear guideline for the model in the prompt.
18:34
So you try those prompts and then you see,
18:36
look at the,
18:37
and then maybe you can try to add more examples and then just go through,
18:42
get really good response.
18:42
Like try to evaluate,
18:44
maybe have a,
18:44
create,
18:45
a set of queries and expert responses using both automated metrics as a judge
18:47
and also human evaluations to measure the progress.
18:50
So you have done prompting or we have added more examples just as a prompt and
18:54
maybe you will start make it more complex.
19:11
So may use,
19:11
you can give then the model more context so they can answer better questions.
19:13
So maybe when users ask the questions,
19:15
you can have the model pull out all the documents or all the job listings
19:18
related to this question or information about the company,
19:21
get a candidate resume.
19:29
SPEAKER_1: So you build a system, you augment the context with the documents.
19:31
SPEAKER_1: So that's a RAG pattern.
19:33
SPEAKER_1: So I do think it's like RAG is a very, very powerful pattern.
19:38
SPEAKER_1: It's nothing really, really fancy.
19:42
And something interesting about RAG is that a lot of people equate RAG with
19:47
vector search.
19:48
When I see a lot of people like,
19:49
"Oh I want to use RAG and here's,
19:52
likewhatdatabaseIshoulduse?"
19:57
SPEAKER_0: So people jump straight to vector search.
20:00
SPEAKER_1: Yes.
20:00
SPEAKER_1: Yes.
20:05
SPEAKER_0: So as engineers who are like, "I need a vector search database."
20:06
SPEAKER_0: A vector search database.
20:11
SPEAKER_1: Yeah, people love databases, very interesting.
20:14
But I feel like there was the first solution is probably not like
20:17
jump-stretchembeddingbaseretrievalbecauseyouhavetouse-
20:20
SPEAKER_0: Oh, interesting.
20:22
SPEAKER_1: Yeah, you need to build, it's like an embedding model.
20:24
SPEAKER_1: The qualities that really highly dependence on its equality embeddings.
20:28
SPEAKER_1: If you have embeddings numbers, you retrieve things well.
20:30
SPEAKER_1: Also, very databases can be quite expensive to run and a latency.
20:33
Also,
20:34
another thing is vector databases can,
20:35
embeddings can also obscure certain keywords.
20:37
For example,
20:38
if I'm searching for a specific arrow code,
20:39
right,
20:40
through embedding,
20:41
you don't really get the exact error code anymore.
20:49
SPEAKER_1: So there's certain challenging with vector databases and vector search.
20:52
SPEAKER_1: So I think it's like the usual common approach.
20:55
SPEAKER_1: Maybe just start with something as simple like keyword retrieval.
20:59
If you ask,
20:59
just extract all the keywords from the user query and find all the document
21:03
that with the query and you say,
21:04
okay,
21:04
maybe the documents are too long and now it can't fit into context,
21:08
that's when it's start with chunking.
21:09
Okay,
21:10
now how do I chunk this documents into context that can fit into the context
21:14
link and now it's chunking have other problems.
21:16
So for example,
21:16
maybe the keyword,
21:17
the document is about the company X,
21:19
but people say from now on company X is referred to as company,
21:22
so it is a rest document.
21:23
SPEAKER_1: You don't have X appear anywhere.
21:24
SPEAKER_1: So you don't get the chunk.
21:26
SPEAKER_1: If you search for X, you don't get the chunk below.
21:42
So now you might want to,
21:45
okay,
21:45
now I need to extract the keywords from documents and get X metadata to every
21:48
chunk or get the title of the document or some people started adding summary
21:52
to it.
21:52
SPEAKER_1: Like Anthropic has a very, very good article called Contextual Retrieval.
21:57
It's like as they asked ChatGPT to generate the key information metadata
22:00
about each document and append it to prepare that to each chunk to have you
22:07
retrieve the right chunk.
22:08
So I do think this having data preparations,
22:09
right,
22:10
actually give a really huge performance boost.
22:13
I have seen this giving way,
22:14
way better performance boosts and focusing on which better databases I
22:17
should use.
22:25
So,
22:25
what is a little bit of...
22:26
I'm not saying that they're not useful,
22:28
it's just in the beginning you probably wants to try something simple with
22:30
the biggest performance against and then you started moving up the
22:33
complexity level.
22:34
Also for a lot of retrieval,
22:35
someone told me very interesting,
22:38
a little bit of heartache,
22:40
but he was just like,
22:41
"I'm not going to take any retrieval system seriously if they don't
22:44
benchmark against BM25.
22:52
SPEAKER_1: So BM25 is a pretty old school, like 20+ years now.
22:55
SPEAKER_1: Retrieval is a lot simpler, like turn-based retrieval not embedding.
22:59
And it's a really,
22:59
really hard-to-beat retrieval system and I think a lot of time we use it,
23:04
if you start a complex,
23:05
you can now combine both turn-based retrieval and a simpler solutions with
23:09
embedded databases.
23:11
So you have both the semantic on the embedding side,
23:14
but you have [inaudible 00:23:18] the exact keyword match on the turn-based.
23:16
SPEAKER_1: So a lot of things they hybrid search is very common.
23:25
So I guess we talk about prompt engineering,
23:28
add more examples,
23:29
RAG,
23:29
right?
23:30
I think after that maybe have any max out on a lot of those things,
23:34
which usually take why people maybe then it might consider fine-tuning,
23:38
but I think they usually have a lot of reservations against fine-tuning
23:42
because fine-tuning has a whole host of new problems that you need to deal with.
23:46
Like fine-tuning first you need to think of now you have this model,
23:50
you have fine-tuning and now you need to think of how you host it.
23:53
SPEAKER_1: And a lot of models are big, they're building the parameters.
23:56
SPEAKER_1: They are not easy to sell.
24:03
SPEAKER_0: I actually read this part in the book.
24:04
You actually go into the details of the problems with the memory size and also
24:09
you cover the alternatives where you might not need as much memory,
24:12
but they also bring other trade-offs.
24:14
SPEAKER_0: So it's trade-offs within trade-offs within trade-offs, right?
24:18
SPEAKER_1: Yeah.
24:19
You're starting to solve one problem,
24:21
but you're going to get a bunch of others and you'll need to decide is it worth
24:24
my time,
24:24
effort,
24:25
resources.
24:28
SPEAKER_1: Yeah, yeah.
24:28
So I definitely think that...
24:29
also when you find the model,
24:31
you kind of own the fashion model.
24:34
SPEAKER_1: And now the question is how do you maintain it?
24:36
Like,
24:36
anything,
24:36
we have this whole world of very smart people having new models like this
24:39
capability increases rapidly.
24:42
So how long can the fine-tuning model outperform those new models that we're
24:46
putting out?
24:47
So you might spend a lot of energy in a [inaudible 00:24:55] model and just a
24:51
[inaudible 00:24:56],
24:51
maybe like some random Chinese company you never heard of release is
24:54
extremely fast and extra model,
24:57
right?
25:02
SPEAKER_0: Oh yeah.
25:02
SPEAKER_1: So yeah, it is quite challenging.
25:03
So yeah,
25:03
five-tuning,
25:04
you think it is the last resort,
25:06
not the first line of defense.
25:12
Yeah,
25:12
but what I've heard is basically,
25:14
if I got it correctly,
25:15
do a structured approach,
25:18
start with prompting,
25:20
start with simple,
25:22
start with you getting responses that make sense,
25:24
then add more data.
25:25
You can do this with RAG,
25:27
you can do it with chunking,
25:28
keyword extraction,
25:30
data preparation really,
25:31
really makes a big difference,
25:33
which a lot of people don't think about.
25:35
SPEAKER_0: And then you can go to more advanced things.
25:37
There's a whole host of things that you could do,
25:41
but my understanding is you're probably saying you'll probably get there
25:44
over time,
25:44
but initially these things will keep you busy,
25:46
and you'll probably be able to build a pretty good system just with the basics
25:49
and a little bit of engineering,
25:51
and most importantly understanding the problem that you're trying to solve
25:55
as opposed to building whatever shiny technology or approach.
26:01
Yeah,
26:01
I feel like the approach can also be different for individual developers or
26:06
enterprise if you look as a whole organization.
26:09
So one thing I haven't seen is,
26:10
especially in the early day technologies,
26:12
usually enabling new use cases actually bring more returns on this
26:14
incremental improvement of use cases.
26:16
So maybe if instead of spending the effort into investing a lot of energy,
26:22
[inaudible 00:26:28],
26:23
like little bit of performance with fancier complexity,
26:25
maybe using the same suspect you have had and opening up new applications.
26:29
So,
26:30
I said that's why I feel like a lot of companies will take a while to get to the
26:33
fine-tuning phase.
26:45
SPEAKER_0: Trust isn't just earned, it's demanded.
26:46
Whether you're a startup founder navigating your first audit or seasoned
26:49
security professional scaling your governance risk and compliance program,
26:51
proving your commitment to security has never been more critical or more complex.
26:54
SPEAKER_0: That's where Vanta comes in.
26:54
Vanta can help you start or scale your security program by connecting with
26:57
auditors and experts to conduct your audit and set up your security program quickly.
27:00
Plus,
27:00
with automation and AI throughout the platform,
27:02
Vanta gives your time back so you can focus on building your company.
27:04
Businesses use Vanta to establish trust by automating compliance needs
27:09
across over 35 frameworks like SOC 2 and SO 27001.
27:10
With Vanta,
27:10
they centralize security workflows,
27:11
complete questionnaires up to five times faster,
27:13
and proactively manage vendor risk.
27:14
Join over 9,000 global companies to manage risk and prove security in real time.
27:17
For a limited time,
27:19
my listeners get $1,000 off Vanta at vanta.com/pragmatic.
27:21
SPEAKER_0: That is V-A-N-T-A.com/pragmatic for $1,000 off.
27:46
So,
27:46
let's say that at my company we decide to build an AI solution and let's take
27:49
the example of that it's going to be customer service automation.
27:55
SPEAKER_0: What are typical approaches that I should know about?
27:58
SPEAKER_0: And you do cover some of these things in your book as well.
28:02
SPEAKER_0: Some of the kind of more common steps that I'll need to take.
28:05
SPEAKER_1: So customer support solutions.
28:06
So I would say the first thing I would look into is what are the bottlenecks for
28:10
the solution right now?
28:10
So for example,
28:11
I have worked with another startup I work with,
28:13
it actually has this challenge,
28:14
oh,
28:14
they had a lot of customer support requests and they don't know what to answer.
28:18
SPEAKER_1: And the solution is very interesting.
28:20
It's just like,
28:21
okay,
28:21
let's try to drive a lot of questions into the common channel like public discourse.
28:24
SPEAKER_1: So that means, "Are the users going to help answer the questions?"
28:27
So as much,
28:28
and also in the future someone have a question,
28:31
they can just refer to previous discussions instead of asking.
28:34
So they try to make a lot of the discussion customer support request public.
28:48
So another solution was pretty popular in 2018,
28:48
2019 is that you do routing to the right department.
28:51
SPEAKER_1: So the challenge then is they realize the bottleneck is in triaging.
28:54
SPEAKER_1: So first we get a request, I don't know which department to send it to.
28:57
So there were a bunch of startups then,
28:59
it's like,
28:59
"Okay,
28:59
let's try to build a system to predict,
29:01
like should this go to the finance department or should it go to a technical
29:04
supportdepartment?"
29:05
SPEAKER_1: Just like routing already reduce the friction a lot.
29:07
SPEAKER_1: And if you think it's like okay, we need gen AI to do it.
29:10
So I think it really recommend the frameworks that Microsoft introduced
29:12
they call is like CRON work/run going from a slightly lower stake to higher
29:15
stake deployment.
29:41
SPEAKER_1: So first off the customer support chatbot, right?
29:43
SPEAKER_1: So maybe initially you can have a human in the loop.
29:45
So for example,
29:46
you can have for every request instead of a human agent writing the response
29:51
from scratch,
29:51
you can have AI suggest a few options and the human can choose one or can just
29:55
choose ones as a starting point and then make short a quick edit and send it.
30:06
So once you see this,
30:07
okay,
30:07
maybe the acceptance rate is getting really high.
30:10
So first of all,
30:10
for these categories of queries,
30:12
maybe the acceptance rate is 90%,
30:13
so you might feel more confident to roll it out.
30:14
So maybe roll it out to maybe smaller users or you can run it out in even
30:17
internal use cases and so then,
30:18
you give it more automations but reduce the scope of deployment and then
30:21
after you're really happy with it,
30:23
you can run out to more users.
30:25
So yes,
30:25
that's how I would go about something like customer support chatbot.
30:43
Nice,
30:43
because what I was kind of expecting you might say is like,
30:48
"Oh,
30:48
just build this AI framework,
30:49
deploy."
30:49
SPEAKER_0: I feel that's what a lot of companies are doing by the way.
30:52
SPEAKER_0: A lot of things is like, "Oh, gen AI.
30:55
Oh,
30:55
let's get this model from ChatGPT or Anthropic,
30:58
let's try to put it there.
30:59
SPEAKER_0: Let's put it out there."
31:00
But I really like how...
31:01
what you're saying sounds like it's not really,
31:03
what you explained,
31:05
it's not really specific to gen AI that you went through.
31:08
Look at the business problem,
31:10
look at the problems you have,
31:10
look at the options which include not just gen AI but more traditional
31:14
machine learning,
31:16
as you said,
31:17
"Classifier."
31:20
And then you'll look at if these tools help your problem and then you'll make
31:22
sure that it actually solves your problem when you roll it out,
31:25
don't just blindly roll it out.
31:26
SPEAKER_0: Which, I mean, all of this sounds to me it's not really new is it?
31:29
It's not...
31:30
like,
31:31
you could have said the same thing two or three years ago before gen AI,
31:35
except we would have not had those gen AI tools to play with.
31:40
SPEAKER_1: Yeah.
31:40
So I think actually before our chat,
31:41
I should look up one of the talks before and you have this talk about things
31:46
that haven't changed,
31:47
things that feel very similar about social engineering and I definitely
31:50
like dealing with a new technology is one of the things that never change.
31:54
So I feel like every time there's a new technology that comes out,
31:57
I can hear the collective sigh of senior engineers everywhere saying,
32:02
"Noteverythingisnew."
32:04
SPEAKER_1: People just try to get such technology to work for everything.
32:07
So I do think that,
32:07
a very common challenge I see is that people just jump straight into it.
32:11
SPEAKER_1: They just want to use gen AI for when they don't need gen AI.
32:18
SPEAKER_1: So, I think there's two different headlines.
32:20
So one headline is like,
32:21
"I use gen AI," or as the other headline,
32:23
"Isolvedtheproblem."
32:24
So if you want to focus on the first headline here,
32:27
you use gen AI,
32:27
but if you want to solve the problem then you need to understand what is a problem,
32:31
what are the challenges there,
32:32
what's the roadblocks?
32:33
And remove the roadblock using the simplest solutions,
32:35
not the fanciest one.
32:44
Yeah,
32:44
I feel that there's a bit of a really strong fear of missing out across most
32:46
tech companies that everyone knows that this is such a transformative technology.
32:49
SPEAKER_0: It gives us so many new capabilities that it will be important.
32:51
I think everyone knows that their company will be using it,
32:54
but now there's a fear of missing out of,
32:56
"Oh,
32:56
what if my team doesn't build it?
32:58
SPEAKER_0: What if someone else gets ahead of me?"
33:00
And so a lot of companies,
33:01
many teams are all building it and trying to just,
33:03
using a hammer,
33:03
looking for nails even though they might not need it at the time.
33:05
I'm not sure if this is a bad thing necessarily because people at least get
33:07
experience with it and they will need to learn about it.
33:24
But it's a very interesting time,
33:26
because rare to see,
33:27
usually we see a new backend framework come out or something that's limited
33:30
to the main and people jump on it.
33:33
But this is one...
33:34
the first time I've seen that the whole industry is jumping on it and everyone
33:36
is trying to use it and put it in whether it works or not.
33:44
SPEAKER_1: I definitely agree with you on this FOMO thing.
33:46
I do think this is like,
33:48
everyone's jumping on,
33:49
it's actually a pretty good thing.
33:51
SPEAKER_1: I feel like the energy is incredible.
33:54
SPEAKER_1: I've never before seen so many smart people focusing on the same problem.
33:58
SPEAKER_1: It's incredible and the progress is amazing.
33:59
I do think,
34:00
however,
34:00
I think there's an irony is just like the more we want to not miss out on things,
34:04
the more things we will miss,
34:06
because what I think is feel like if we try to keep up with news,
34:09
we should try to keep from jumping from one piece of news to another,
34:12
we will always stay at the surface level.
34:14
SPEAKER_1: We never really go deep into anything.
34:22
So I actually don't quite read news,
34:23
so I find like a little bit distracting.
34:24
So I think,
34:24
I try to,
34:24
feel like my approach is like,
34:25
okay,
34:25
pick a problems that you care about and then only care about things just like
34:28
how you solve this problem.
34:29
So if there's some news coming out and was just like,
34:31
"Does this help me solve this problem?
34:33
If it doesn't,
34:33
I can't wait," because I feel like if there's something important,
34:37
it will still be important two weeks from now on,
34:38
a month from now on.
34:39
Like I don't drop everything and it's like,
34:41
okay,
34:42
let's just go and understand what it is.
34:44
SPEAKER_1: So I feel like trying to get a more, try to stay bit calmer, yeah.
35:05
So when you're building an AI system,
35:06
one of the things that you will come across is you need to evaluate the output,
35:09
how well it works,
35:10
does it solve your problem.
35:11
Why is it difficult to evaluate AI systems and what are common ways to do that?
35:23
So evaluation I think is billion-dollar question or even a trillion-dollar
35:26
given how much we're investing right now.
35:30
SPEAKER_0: Trillion-dollar, nice.
35:31
SPEAKER_1: Like, we need to go big and if go big, better go really big.
35:34
So I think this it's challenging because the smarter AI becomes,
35:39
the harder it is for us human to evaluate it.
35:42
So before,
35:43
if AI was incoherent,
35:43
you can pretty tell if the response is bad,
35:46
I say,
35:46
"Okay,
35:46
doesn't sound good,
35:47
it'sabadresponse."
35:47
SPEAKER_1: But nowadays, it's pretty coherent.
35:50
SPEAKER_1: First of all, you ask, should I generate a summary of a book?
35:53
If the summary sounds convincing,
35:55
you actually don't know if it's a good summary or not and you might have to read
36:00
the entire book yourself just to evaluate whether it's a good summary.
36:13
The math,
36:13
a lot of time I personally use AI,
36:15
actually to ask a lot of questions because I don't know the answer and because
36:19
I don't know the answer,
36:19
I don't know if the answer is correct,
36:22
right?
36:22
So,
36:22
an example was a lot of people can tell if a math solution should,
36:25
a first grade question is correct,
36:26
but very few people can tell is if a fancy equation or proof is correct.
36:40
So I remember when Avant came out,
36:42
Terence Tao,
36:42
he's this amazing mathematician,
36:43
I think it's one of the best mathematicians of our time,
36:45
he actually took time to evaluate Avant and he says the experience of using
36:49
Avant is similar to advising an incompetent but not completely stupid,
36:54
like a mediocre but not completely incompetent student.
37:01
But that makes me think,
37:02
if we really need the brightest minds today to evaluate AI,
37:05
then we are soon to run out really,
37:06
really smart people to evaluate AI.
37:18
SPEAKER_1: So what would be the next step forward?
37:19
So before a lot of time we use a whole human as a gold standard for AI performance.
37:24
It's like okay,
37:24
so humans starting writing out,
37:25
"Here is how you should respond to this and here's how you should do it," and AI
37:30
should try to copy human.
37:32
SPEAKER_1: But now for many, many tasks we have AI outperform human way better.
37:37
So I think there's several...
37:40
so I thought about several approach to deal with it and I think that's why I
37:43
separate the chapter like those.
37:54
SPEAKER_1: So, initially I had one chapter on evaluation.
37:56
SPEAKER_1: There's more I write about it.
37:58
It was like should...
37:58
there's so much,
37:59
enough,
37:59
two pretty long chapters on evaluation.
38:00
The first chapter is on general methodology and the second chapter is about
38:03
how to use different techniques to evaluate the AI system.
38:06
SPEAKER_1: So as methodology, one is that functional correctness.
38:09
So it evaluates the output of application based on how well it performs a task.
38:13
So if you say that,
38:13
"Hey,
38:13
use AI to save energy," you can see how much energy is actually saved.
38:15
Or,
38:16
"Hey use AI to play this video game," you can see how high is the score you can
38:19
actually get.
38:20
SPEAKER_1: Or, "A very common use case for this coding."
38:37
SPEAKER_1: So I do think it's not a code incident, it's a coding.
38:39
It's one of the most popular use cases because we actually know how to
38:44
evaluate generated code.
38:45
We might not know to evaluate generated essays and stuff,
38:47
but you know how to evaluate generated code because we've been testing code
38:54
for a long time.
38:55
So with code you can do use functional correctness to evaluate first whether
38:59
is this code comply,
39:00
does it run?
39:01
SPEAKER_1: Second, does it generate expected outputs?
39:03
SPEAKER_1: That's what we wanted to do.
39:05
SPEAKER_1: So that's one approach.
39:08
SPEAKER_1: The second approach is using AI to evaluate other AI.
39:11
SPEAKER_1: So we have been using AI to evaluate, to automate a lot of applications.
39:17
SPEAKER_1: So can we also use AI to automate evaluations?
39:20
SPEAKER_1: And actually they're doing pretty well.
39:22
I think we have this like...
39:23
I think,
39:23
in many,
39:24
many,
39:24
even back in 2023,
39:24
[inaudible 00:39:30] has this report.
39:26
They saw as the majority of applications they saw already has some sort of AI
39:29
as a judge or [inaudible 00:39:38] as a judge.
39:32
SPEAKER_1: And I think it's growing.
39:33
SPEAKER_1: And we do things that it's getting pretty cost affections and useful.
39:37
But,
39:37
of course,
39:38
I see a lot of challenges around using Azure just so we can go into later.
39:51
But another approach I thought was very interesting is on comparative evaluations.
39:56
And the reason,
39:56
as humans it might be hard for us to give an absolute score on something,
39:59
but if we can give two versions of something,
40:01
we can tell,
40:02
"Oh,
40:02
werealizethisoneisbetter."
40:03
So we have done a lot of studies showing that even for tasks where AI is outperforming,
40:08
doing as a level with humans,
40:09
experts can't really do,
40:11
we can still tell,
40:12
detect the differences.
40:13
So I think this hasn't been a lot,
40:14
been guiding,
40:16
not just evaluations but also model development.
40:31
SPEAKER_0: Sounds like there's just no simple answer, right?
40:32
You kind of need to go through all these options and figure out in your case,
40:38
which makes sense for costs for what you can do,
40:40
can you have a human in the loop so there's no real silver bullet,
40:42
no one thing that you can just use.
40:51
SPEAKER_1: Yeah, I don't think there's a simple solution.
40:52
So that's one thing I'm a little bit skeptical about,
40:55
evaluation tooling,
40:56
because a lot of challenge evaluations,
40:58
or not because we don't know how to evaluate,
41:00
but because it's required disciplines and hard work and a lot of things is
41:03
like tools can't really automate.
41:05
So,
41:05
for example,
41:05
one thing for evaluation,
41:07
we need to evaluate an application based on what users want.
41:11
And so that means that we need to go and talk to users,
41:14
we need to look at their interactions,
41:17
because a lot of things what we think is like we want you to evaluate what matters.
41:23
SPEAKER_1: We have to measure what matters.
41:38
So for example,
41:38
I have several examples of how it's very counter-intuitive thinking that we
41:41
are measuring one thing,
41:42
but user actually care about other things.
41:45
So in the beginning,
41:46
for example,
41:46
I have a friend who building a pretty big applications basically building to
41:50
summarize meetings.
41:51
SPEAKER_1: And initially they were like, "Okay, we try to measure correctness.
41:54
SPEAKER_1: Does the summary cover the content of this meeting?"
41:57
SPEAKER_1: I say, so think about, "Hey, does the model follow the format?
42:00
Because they think the site users want a shorter summary and they agonize over,
42:04
dowewantthree-sentencesummariesorfive-sentencesummaries?"
42:07
SPEAKER_1: And they try to measures though.
42:09
But actually eventually what they found out that users don't really care
42:11
about the whole content of the meeting.
42:14
People only want,
42:15
"What is the action item for me,
42:16
likewhatwehadtodoafterthis."
42:32
SPEAKER_1: So they start changing.
42:33
SPEAKER_1: So they don't measure correctness anymore.
42:34
I mean they just don't make up things,
42:37
but they focus on,
42:39
don't miss out on action items.
42:40
SPEAKER_1: Best fit for the person asking for the summary.
42:43
So I was like...
42:45
or the example is some people using chatbot for,
42:47
so we talk about a customer support chatbot.
42:49
SPEAKER_1: So I want to go back to that example.
42:53
SPEAKER_1: So a pretty big tax firm.
42:54
So it'd be with chatbot,
42:58
so tax software,
42:59
you can pretty much tell which company it is.
43:03
SPEAKER_1: So they just launch a chatbot to help people with tax preparations.
43:07
SPEAKER_1: And the responses were very lukewarm.
43:11
SPEAKER_1: They were measuring by the users, how they use it.
43:14
SPEAKER_1: And it was like people just didn't really seem to use it.
43:17
SPEAKER_1: And they were like, "Why is that?
43:19
SPEAKER_1: Is that because it's not, what hallucinate, or what is the challenge?"
43:26
So they charge you to measure all those kind metrics,
43:28
but in the end,
43:29
user didn't use it because seems they hate typing.
43:31
SPEAKER_1: People don't really like typing.
43:34
And also because in your face with a domain,
43:36
a domain that you don't really know,
43:37
I use the software because I don't know a lot of things about tax.
43:40
SPEAKER_1: I never know what question to ask.
43:45
SPEAKER_0: Oh, so they just didn't know what to type.
43:46
SPEAKER_0: They didn't understand the domain.
43:49
They went to this tax thing because they wanted to take care of their tax.
43:57
So I think myself started trying to understand more what type of questions
44:02
people would ask and suggest that in the beginning and then basically say
44:07
guide users.
44:08
So it's kind of education,
44:10
"Here's a question you should ask and then here's the answers,
44:12
keepgoing."
44:13
So I think a lot of that is it's just a lot of understanding your domain as a
44:16
problem domain,
44:17
like go talk to users,
44:17
looking at the data.
44:19
SPEAKER_1: I do still think that looking at data is very, very important.
44:21
SPEAKER_1: I think Greg Brockman has a great quote about it.
44:24
"So you think it's manual data inspections is orders of activities that has a
44:28
highestratioofvaluetopackage."
44:40
So that means that's people don't think highly of manual data inspection of
44:46
data wrangling.
44:47
Let's give some interns to do,
44:48
let me think of something fancy or algorithms and stuff.
44:53
But actually it's extremely high value because by looking at data you detect patterns,
44:58
you understand how that users use a product.
45:01
So actually I usually a very good practice,
45:02
I really highly recommend to teams.
45:04
SPEAKER_1: It's like don't forget human evaluations.
45:07
So you use ASA judge,
45:08
but as a judge has a lot of challenges because the quality of the judge depend
45:12
on the underlying model and the prompt and also non-deterministic.
45:17
SPEAKER_1: So things can change over time.
45:19
But if you have some human evaluations,
45:20
very consistent,
45:20
very clear guidelines,
45:22
every day go in there,
45:24
look at maybe 50 samples of actual interactions.
45:26
Or if you have more resources,
45:27
go as high as like 500 or 1000,
45:28
so that you can get some kind of picture of how first,
45:31
how the users are using your product,
45:32
any changes in behavior based on current events,
45:34
maybe because of recent administration change,
45:35
maybe we'll have a lot more questions about that topic for example,
45:39
or help you correlate with all the automated metrics.
45:41
First of all,
45:42
if the AI just score somehow start changing compared to the human's judge score,
45:45
maybe there's something you to investigate.
46:07
Yeah,
46:07
so I guess you cannot,
46:10
as you said,
46:12
you can't really skip hard work if you want to get good results and you can't
46:17
really pull humans out of the loop fully,
46:20
at least initially.
46:21
What are some common mistakes you've seen when teams are building AI applications?
46:26
SPEAKER_1: Common mistakes.
46:31
SPEAKER_0: I'm sure you did a lot.
46:33
Yeah,
46:33
but I feel like I don't want to say otherwise everyone say she is an idiot.
46:38
SPEAKER_1: Yeah, I think we touched on several.
46:39
SPEAKER_1: So one very common mistake is like use gen AI when you don't need gen AI.
46:42
SPEAKER_1: So first of all, this is a startup that came with me with a pitch.
46:46
It was like,
46:47
"Oh,
46:47
we'regoingtousegenAItohelppeopleoptimizeelectricityusage."
46:51
So when I asked,
46:53
"So people can tell the chat,
46:56
the AI,
46:56
'Hey,
46:56
I live here and here are the activities I do during the days that are very
47:02
energy intensive,
47:02
maybeinchargingacarordoinglaundryorsomething.'
47:07
And AI is going to tell you,
47:09
'Hey,
47:09
you should do this activity this time and this time so that you can minimize
47:12
theelectricitybill.'"
47:13
And they were like,
47:14
"Oh,
47:14
ourreasonsshowthatifcansaveyouonaverage30%ofelectricitybill."
47:18
SPEAKER_1: It was like free money.
47:20
SPEAKER_1: Why would anyone not want that?
47:22
So I was asking them what is this cost saving if I'm just manually schedule the
47:26
most intensive one during the off-peak hours and it just,
47:29
it'll go like,
47:30
okay,
47:30
just charge the car at 10 p.m or something.
47:32
They were like,
47:33
"We haven't done that yet,
47:34
butwearegoingtotryitandletyouknow."
47:36
SPEAKER_1: And they never got back to me and they abandoned the idea later.
47:39
So I feel like a lot of those optimization problems can be solved like readily,
47:42
maybeevenaspreadsheet-
47:56
SPEAKER_0: Without gen AI?
47:57
SPEAKER_1: Without gen AI.
47:57
Another spectrum is that I see a lot of companies giving up on gen AI because
48:03
they think that gen AI is not good for that problem because they have tried it
48:06
and it doesn't work.
48:07
SPEAKER_1: And a lot of time I got surprised.
48:08
I was like,
48:09
"Wait a second,
48:10
I just talked to another company who just,
48:12
theyusethatforthesimilarusecaseanditworkedreallywell."
48:19
And when we look into it,
48:21
usually because of bad product,
48:21
because they don't promise well,
48:23
they don't understand the users,
48:25
they don't even know what to evaluate well.
48:28
So first of all,
48:29
I was working with these companies that does basically extracting resume information.
48:34
So there's a person,
48:36
I get a resume and it tries to map out where does the person work before and
48:40
create a summary of that person's life.
48:42
SPEAKER_1: And they have two steps.
48:43
First from resumes they try to extract on the text and then after own the
48:47
extracted text,
48:47
they extract the organizations from the extracted text.
48:50
SPEAKER_1: So by the way, the resume is a PDF, not pure text, right?
48:53
SPEAKER_1: And then I asked them... so it worked terrible.
48:55
SPEAKER_1: They got the organizations wrong about 50% of the time.
49:11
SPEAKER_1: And then I was asking them, "Where in the process does this fail?
49:13
Is that in from the PDF to extracted text or from the extracted text
49:17
organizationextraction?"
49:18
SPEAKER_1: They were like, "Oh, we don't know.
49:19
SPEAKER_1: We didn't do that."
49:21
It was like if you can't pinpoint,
49:23
you can't localize where it fails then how can you fix it?
49:27
So a lot of times it seems like common sense,
49:28
but somehow,
49:29
I don't know,
49:29
it's just something is always puzzle me a little.
49:38
Or another is just statue complex for first of all,
49:40
jump [inaudible 00:49:48] bases or fine-tuning.
49:41
Or another common one nowadays it's like when you see a fancy agent framework,
49:44
just let's use this framework,
49:45
let's just try it.
49:46
SPEAKER_1: And I think that eventually attractions are really, really cool.
49:49
SPEAKER_1: I think very grateful for many attractions that make my life easier.
49:52
But I do think that the attraction should encode best practices and should be
49:55
heavily tested.
50:13
But I think we still in the phase,
50:14
we're still learning best practices and also a lot of attractions can
50:19
introduce unnecessary very,
50:20
very painful bugs.
50:22
So when I was going through the code basis of a lot of those frameworks and I
50:26
found out something interesting.
50:29
A lot of those frameworks are some deformed prompts you can have,
50:33
you get started,
50:33
because it just make very easy for you to begin.
50:35
But then every single of those prompts and look have some type typos and
50:40
they're just like chains.
50:42
So you have somebody submit a quick PR to fix the typos,
50:44
but it's not part that release anything.
50:47
So if you're using this framework,
50:50
using one of the deformed prompts,
50:52
and then suddenly the performance applications just change,
50:54
eventually don't quite know why is that changing?
50:57
SPEAKER_1: Because prompt was changing under the feed.
50:59
SPEAKER_1: So those are very interesting.
51:00
SPEAKER_1: Those are just patterns that I've seen.
51:14
It's interesting,
51:15
because what you mentioned,
51:17
it sounds like if I'm collecting these,
51:18
it's like using this technology when you don't really need it,
51:21
giving up on it without just for common sense reasons.
51:24
You could have just fixed some easy things,
51:26
using a new framework when it's just not really high quality,
51:29
and it doesn't really have the best practices.
51:32
This kind of all sounds stuff that we could just replace gen AI with a new
51:37
technology or a new stack and we will probably hear similar things,
51:40
right?
51:40
It's typically these things,
51:41
because it's changing all the time,
51:44
there's no best practices,
51:45
no one really knows how to use it.
51:46
Whoever tells you they're the expert,
51:48
they're still just,
51:49
they have a maximum year of experience with it.
51:51
SPEAKER_0: It's not really new, is it?
52:04
SPEAKER_1: Yeah, I think I definitely agree with you.
52:06
I do think that even though technologies change over time,
52:09
so systematic thinking,
52:11
it's like systematic approaches to problem usually don't change.
52:13
If you want to solve problems,
52:15
you first start by breaking down the problems,
52:18
seeing where the challenges are and go through different solutions to do that.
52:21
SPEAKER_1: It seemed common.
52:22
But I think a lot of time,
52:24
a lot us get formal,
52:24
I think formal get in the way and was like,
52:26
"Okay,
52:27
we know this is the right thing to do,
52:28
but I also feel like I just need to check this thing out first and keep doing
52:33
thatthreetimesaday."
52:35
So day's gone and you just never really get time to sit down and think really
52:39
deeply about what is that you're trying to do.
52:55
Yeah,
52:55
so I guess we're going to see a lot of the mistakes that are with new technology happen.
52:58
Plus if some of the listeners have adopted new technology,
53:00
you can probably use some of that approaches.
53:02
I mean just localize it for gen AI and see if you can avoid some of those.
53:05
Speaking of new technology,
53:06
as someone who's learning gen AI,
53:07
a software engineer who wants to get into AI engineering,
53:09
what would your recommendation be to learn,
53:12
if things do change so fast?
53:13
SPEAKER_0: You did mention the importance of fundamentals.
53:16
SPEAKER_0: What would you focus on?
53:27
So I have a lot of thoughts on learning because I like learning a lot and I think
53:32
over time I just try to just observe some patterns.
53:37
And by the way,
53:39
the way I learn things might not be the same as the way you're learning.
53:43
SPEAKER_1: People have different learning style.
53:45
SPEAKER_1: But in general, I think of learning has two different approach.
53:46
SPEAKER_1: One is project-based learning and the other is structured learning.
53:49
Project-based learning is like,
53:50
okay,
53:51
you choose a project and you work on it,
53:53
really go and try and solve every problem in that project and finish it.
53:57
But structure-based learning is more like when you take a course or you read a book,
54:03
it's just like somebody else laid out,
54:05
here's the things that you want to do.
54:07
And I think there's quite a bit of a debate on...
54:10
somebody told me recently that,
54:12
a friend,
54:12
a very good friend,
54:14
and he said he thinks it's a problem nowadays with people who want to become engineer,
54:20
is that they spend too much time learning and not enough time doing.
54:29
SPEAKER_1: And he was "Just forget all the courses, forget all the books.
54:32
SPEAKER_1: Just pick a project and just work on it."
54:33
SPEAKER_1: And I do think this project-based learning is very, very valuable.
54:35
But you think of here is a set of the skills and knowledge I want to do,
54:39
I need to have to become really good at something.
54:41
Project-based learning can have you hit a lot of these points,
54:44
but it doesn't always have you hit all the points and it can sometimes get the confusions,
54:49
whereas also sometimes students to complement the structured learning.
54:57
And another thing that struck my project-based learning that a lot of people
55:01
usually do it follow some tutorials.
55:03
SPEAKER_1: People like here someone has this picture how to do this.
55:07
And I think tutorials is really cool and I think I'm so personally do it a lot,
55:10
but I still notice that it's very easy choose just mindlessly clicking one
55:15
cell to another and just run the cell,
55:17
run another,
55:17
and don't really stop to ask why is this being this way?
55:25
SPEAKER_1: Why is this library being important?
55:27
SPEAKER_1: Why is this code written this way?
55:28
SPEAKER_1: Why is the batch size is 16 instead of 64?
55:30
SPEAKER_1: Why is there expert in the mixture of expert?
55:33
SPEAKER_1: So some people, it's very easy to not stop.
55:34
SPEAKER_1: There's no mechanism to force you to stop.
55:36
You just want to run to the end and see what the output and make some changes by
55:39
the best guesses.
55:40
SPEAKER_1: It's something funny.
55:41
Like,
55:41
when I was working on this open source project and it was like,
55:45
it didn't want to do a market research and see who is using this framework.
55:48
So it was a framework with iBiz,
55:50
and I thought it's like...
55:51
I knew that if you wanted to use this framework,
55:53
you need to import iBiz.
55:54
So I went through on a GitHub and I searched for all the repos that like import iBiz.
55:58
SPEAKER_1: And I found repos that I went to it.
55:59
It has import iBiz,
56:01
but then the code base does not have iBiz anywhere else.
56:03
SPEAKER_1: It's not used at all.
56:04
SPEAKER_1: And I was like, "What is happening?"
56:06
SPEAKER_1: So I realized that's a lot of those repos, copy from a tutorial.
56:10
SPEAKER_1: And that tutorial use, import iBiz.
56:28
SPEAKER_1: And that's my mistake.
56:28
SPEAKER_1: And then like everyone else.
56:29
So maybe the original developer import iBiz and then deleted the code
56:33
because they didn't use it anymore and then everyone could copy it just like
56:36
it's the same thing.
56:37
SPEAKER_1: So I feel like that is something a little bit dangerous.
56:39
Tutorial based learning is great,
56:42
but I do think it's very important to be able to stop and ask questions.
56:45
And sometimes structured learning can help you ask the right questions,
56:47
like think things through.
56:48
So yeah,
56:48
I think if before was starting,
56:49
I would recommend maybe a mixture like yes,
56:51
choose some project you want to work on.
56:53
It doesn't have to be big fancy project,
56:54
just try to pick one and then at the same time compliment it with some
56:58
structured learning.
56:59
SPEAKER_1: Pick whatever, maybe a book or doing a course with a friend, read paper.
57:02
SPEAKER_1: I think read paper is a bit interesting because reading paper is a skill.
57:07
It can be quite time-consuming and you need to know what you want to get out of it.
57:32
SPEAKER_1: But yeah, so start a project, compliment with structured learning.
57:34
At the same time there's an exercise that I felt very,
57:38
very useful,
57:39
at least for me initially,
57:40
is that for a week I try to observe what I do and I try to make note of what I do and
57:45
try to think of what percentage of that can be automated by AI,
57:49
what could be done by AI,
57:50
and then I try to use AI to do those.
57:52
SPEAKER_1: And this just gave me a lot of ideas on the use cases.
57:54
Just think about it,
57:54
what matters to me,
57:56
and it'll be an application that can solve the problem.
57:58
SPEAKER_1: Its great already.
58:09
Yeah,
58:10
I think that's an unconventional way,
58:12
but it is a good way to look at it because in the end...
58:15
also I think it might help you get ahead of this dread of what would AI do for me?
58:19
SPEAKER_0: Because you realize what happens when you automate things.
58:22
SPEAKER_0: Which actually leads to my next question.
58:24
There's a lot of fearmongering around,
58:26
oh,
58:26
AI will mean the end of software engineering because AI is very good at coding.
58:28
SPEAKER_0: It's a lot better than a lot of other areas.
58:30
What is your take on as AI gets better will it actually end software
58:34
engineering or it will change it or it's not going to much change actually?
58:45
SPEAKER_1: I think it goes back to the question of what software engineering is.
58:48
SPEAKER_1: So maybe you can get an analogy, maybe you can help explain it better.
58:53
So the writing,
58:54
so we tend to confuse the most salient activity of something as the job itself.
58:58
SPEAKER_1: So, first of all, writing.
58:59
Writing in the past,
59:00
writingmeansasaphysicalactofputtingsomewordsontopaper-
59:11
SPEAKER_0: On a paper or-
59:12
SPEAKER_1: Yeah.
59:12
And back then people think of writing as that and people actually took pride
59:15
in the calligraphy like,
59:17
"Oh,
59:17
you have beautiful handwriting,
59:18
you must be smart,
59:19
youmustbeintelligent."
59:20
But then we had computers and now writing doesn't refer to the act anymore.
59:25
SPEAKER_1: Writing refers a process of arranging ideas into a readable format.
59:28
SPEAKER_1: And I think the same thing is coding.
59:31
So now people think,
59:32
so engineering thinking it is a physical act of putting code on a VS Code or Vim
59:38
or whatever software that you use,
59:40
but that's not what engineering is.
59:42
SPEAKER_1: So engineering is about solving problems.
59:45
SPEAKER_1: Here's the problems.
59:46
SPEAKER_1: How do I come up with executable programs to solve this problem?
59:49
Coding itself is just a physical act of it,
59:52
and I do think it's like,
59:53
yes,
59:53
maybe AI can have you automate coding,
59:55
but I don't think it's going to fully automate problem solving because you
59:58
still need to know what problems is and only you can understand what problems
1:00:03
you're facing.
1:00:11
And also AI has the problem like coding really,
1:00:13
or software engineering really,
1:00:14
is like,
1:00:14
yes,
1:00:15
you need to solve problems,
1:00:17
but what I don't think we say is you need to do it very precisely.
1:00:20
The reason the job software engineer or programmer exists is because it is
1:00:24
very hard to be specific to speed the computer's language,
1:00:27
because you know if you move that "If statement" somewhere else,
1:00:34
or if you change a variable,
1:00:36
suddenly the program crashes because now you have a stack overflow exception,
1:00:40
which you,
1:00:40
of course,
1:00:41
understand if you're a software engineer,
1:00:44
but if this is just a business user who says,
1:00:47
"I want to show if you resize the window,
1:00:50
I want the button to move over," it's easy to say.
1:00:54
But then as a software engineer,
1:00:56
you know the edge cases,
1:00:57
you know the environment,
1:00:58
you know what needed to worry about,
1:00:59
you need to worry about system events,
1:01:02
et cetera.
1:01:04
SPEAKER_0: And then you'd write code for all of those things.
1:01:07
And I'm sure we'll get to a point where the AI will be able to generate some of that,
1:01:09
but it might not.
1:01:10
And you will still at some point need someone who understands that code and
1:01:13
can figure out where the gap is.
1:01:15
Because English as a language is not as precise as a programming language.
1:01:19
Programming languages were invented to be very precise and unambiguous and
1:01:23
very easy.
1:01:24
You can go from assembly code to the programming language,
1:01:27
because it's a one-on-one mapping,
1:01:28
and then from English to a programming language,
1:01:30
that's very fuzzy,
1:01:30
right?
1:01:36
SPEAKER_1: Yeah, definitely.
1:01:39
SPEAKER_0: I think that profession will not go away.
1:01:40
For users it might work for some kind of more hobbyist use cases of like you say
1:01:45
something and you get something roughly that's it,
1:01:47
and you try multiple times and it generates something else and you're happy.
1:01:52
But for a business or a professional use case,
1:01:54
you will need those people who will be able to guarantee that you get exactly
1:01:57
what you want.
1:02:00
Yeah,
1:02:00
I'm actually really excited about AI can automate part of coding because it
1:02:02
actually enables software engineers should do software much more complex.
1:02:05
SPEAKER_1: So I go back to the analogy of writing.
1:02:06
Before when I had to do the manual copying words onto papers,
1:02:08
all the books back then were very small.
1:02:09
I think it's 5,000 words or 10,000 words considered big because it took a long
1:02:12
time for people to copying things.
1:02:13
But now we have books like a hundred of thousand words and I think it just makes
1:02:16
things a lot easier.
1:02:17
SPEAKER_1: I do think software engineering.
1:02:18
If you don't have to manually write code,
1:02:19
it quickly can turn ideas into snippets,
1:02:21
executable programs.
1:02:22
I don't think the [inaudible 01:02:42] running much,
1:02:24
much more complex software.
1:02:45
And maybe one software engineer will be able to command a lot more software
1:02:49
debug or maintain a lot more complex system by oneself.
1:02:51
Because right now there's a reason that for a million lines of code,
1:02:54
usually there's several engineers.
1:02:55
SPEAKER_0: It's rare to have just one engineer.
1:02:57
SPEAKER_0: If a company broke that, I'm not talking about dependencies.
1:03:00
SPEAKER_0: So that will be interesting.
1:03:01
So what other use cases are you excited about for that AI could bring outside
1:03:03
of just coding?
1:03:16
SPEAKER_1: Let's see.
1:03:16
SPEAKER_1: I think I'm excited about education.
1:03:19
SPEAKER_1: So I do thing like AI can help people learn.
1:03:21
SPEAKER_1: We want to learn a lot faster.
1:03:22
So I think one thing I realized is that nowadays if you know the questions,
1:03:26
finding the answer is actually quite easy.
1:03:28
You can ask AI and it usually give you pretty,
1:03:30
a lot of,
1:03:31
at least it can give you a lot of references for you to go and read more about it.
1:03:35
But then what's still hard,
1:03:36
it's how should you come up with the right questions and I think it's like
1:03:39
education needs to focus on forcing students,
1:03:40
like create the habit of asking questions and understanding.
1:03:44
SPEAKER_1: So I do think it help learning to become a lot more efficient.
1:03:46
People can learn a lot of things and I feel like I do believe that if we can learn
1:03:50
better and faster,
1:03:50
then we can actually do more things.
1:03:53
SPEAKER_1: So I'm very excited about what that would look like.
1:04:31
SPEAKER_1: What other use cases I'm excited about?
1:04:32
SPEAKER_1: I'm excited about entertainment.
1:04:33
Yeah,
1:04:33
I think it's sometime we always think of entertainment and education as
1:04:35
separate things,
1:04:36
but I don't see why we can't have games that help to learn things more.
1:04:39
We have some strategy games like teaching girls about negotiation,
1:04:42
so that could be really fun.
1:04:43
SPEAKER_1: Or there's something more intellectually stimulating content.
1:04:46
Movies or shows don't have to be about mindless...
1:04:47
but I know other people watch that because it's help them escape.
1:04:50
But I also think that genres,
1:04:51
it's maybe think a little or understand more about different fields.
1:05:26
So I do think that now AI can assist us in creating content that is both
1:05:31
entertaining and intellectually stimulating.
1:05:33
SPEAKER_1: So I think it could be a lot of fun.
1:05:35
For some boys,
1:05:36
it's like now that we have a lot of medium adaptations,
1:05:38
so you have a book,
1:05:38
you can convert it into a movie and sometimes you have a movie,
1:05:41
you convert it into a game.
1:05:44
SPEAKER_1: Or you have papers convert into a podcast.
1:05:46
So now if we have some content,
1:05:48
AI can help us convert,
1:05:50
adapt different medium,
1:05:51
that could be very,
1:05:53
very exciting.
1:06:05
Yeah,
1:06:05
I think it's...
1:06:06
there are a lot of small problems that I'm also interested in.
1:06:09
SPEAKER_1: I think I haven't touched on any of the enterprise.
1:06:11
SPEAKER_1: I feel like that's where most money is still.
1:06:13
I do think there a lot of...
1:06:15
I think enterprise or company organization structure is going to change.
1:06:19
SPEAKER_1: So first, what does that mean?
1:06:20
If you think about a lot of organizations,
1:06:21
what is the job as a middle management is to first just upgrading information
1:06:26
from the reports and transmit it up to the executives and is the other way they
1:06:31
transport it,
1:06:32
transmitting information directions from executive to lower layers.
1:06:37
But information aggregation is actually really,
1:06:41
really,
1:06:42
and something AI can do really,
1:06:43
really well.
1:06:44
SPEAKER_1: So I do think this, companies can be a lot more efficient.
1:07:07
SPEAKER_0: So let's close up with some rapid questions.
1:07:08
SPEAKER_0: So I'll just shoot some questions and you tell me what pops to your mind.
1:07:12
What programming language did you use most when you built AI applications or
1:07:15
did ML engineering and why?
1:07:20
SPEAKER_1: Python and JavaScript.
1:07:22
SPEAKER_0: JavaScript as well.
1:07:22
SPEAKER_0: How come?
1:07:24
Yeah,
1:07:24
definitely I think it's a huge part of building products and I have to build
1:07:29
demo quickly,
1:07:30
so I think that's very,
1:07:31
very handy.
1:07:32
SPEAKER_1: I'm not very good.
1:07:32
SPEAKER_1: I think I've always been scared of JavaScript, but I'm very grateful.
1:07:35
SPEAKER_1: AI actually helped me getting started a lot easier nowadays.
1:07:42
SPEAKER_0: And which one is your favorite LL model right now and why?
1:07:50
SPEAKER_1: I don't really have a favorite.
1:07:52
SPEAKER_1: I use it for different things.
1:07:53
So I still use ChatGPT out of habit because I have a bunch of prompts that I use
1:07:59
already and I have little things like rebuilds prompts for ChatGPT I'm still using.
1:08:06
I use Clocks sometimes for grid writing because I think it's sometimes less cliche.
1:08:11
SPEAKER_1: I'm reading on DeepSeek R1 and who's not reading about R1.
1:08:14
SPEAKER_1: So just showing up.
1:08:16
SPEAKER_1: I don't think I have a favorite.
1:08:17
I think I did some of Lava,
1:08:20
which is the version of Lama before,
1:08:21
for some kind of interesting use case from a screenshot to code.
1:08:24
SPEAKER_1: Just trying to test this out.
1:08:25
SPEAKER_1: I'm having fun with it.
1:08:26
SPEAKER_1: But yeah, I'm not emotionally attached to any of them.
1:08:40
SPEAKER_0: And what's a neat AI tool that you've used and that you like?
1:08:47
SPEAKER_1: So I have some things that I built that's really helped me do research.
1:08:49
So one thing is when I saw a link,
1:08:50
like of papers,
1:08:51
it's a lot of links and I try to get a...
1:08:53
So I realized when I read the links,
1:08:55
I do through the same process.
1:08:56
I read the abstract,
1:08:57
I look up the authors,
1:08:58
use a bit work,
1:08:59
ask questions,
1:08:59
and also when I was out,
1:09:00
check the citations.
1:09:01
SPEAKER_1: So I have to do the tool just go and scan so I have a link.
1:09:04
SPEAKER_1: Just give me all the information on like a high level-
1:09:19
SPEAKER_0: So that's a tool to scratch your own itch.
1:09:23
SPEAKER_1: Yeah, I think that's the beauty of AI now.
1:09:26
SPEAKER_1: You can build, it's taking very small amount of time.
1:09:28
SPEAKER_1: You just build a tool just for you.
1:09:29
Before it could take me weeks,
1:09:30
but now I can just do one of that,
1:09:31
I don't know,
1:09:32
like in hours.
1:09:40
SPEAKER_0: This is something to be excited about.
1:09:41
SPEAKER_0: I agree with it.
1:09:42
SPEAKER_1: Yeah.
1:09:43
SPEAKER_1: Yeah.
1:09:45
SPEAKER_0: What are one or two books that you've read and would recommend?
1:09:50
Oh,
1:09:50
I recommend a lot of books,
1:09:51
but I feel like recommending books when I feel like forcing people to...
1:09:55
do what you enjoy.
1:09:55
So I like books that's helped me get a new perspective or give me insight into
1:10:00
some topics I don't know a lot about.
1:10:02
SPEAKER_1: So I realized the book, first of all, Complex Adapted Systems.
1:10:06
It's a very interesting book about a system thinking how to design,
1:10:08
like,
1:10:09
how should design social dynamics so that to get people to work toward the
1:10:14
goals that you want to work on.
1:10:16
SPEAKER_1: It's a very interesting book, forced you to think about systems.
1:10:30
I like the book The Selfish Gene because you understand more about free will
1:10:34
and makes you question a little stuff,
1:10:38
but it's the idea of you can live on either through genes or through ideas.
1:10:41
SPEAKER_1: There are two ways that they can live on.
1:10:44
So genes could lead you to live on with their offspring and reproductions.
1:10:49
The other thing is if you have ideas and the ideas can also replicate,
1:10:51
like the one like idea of memes,
1:10:52
like genes and memes.
1:11:00
SPEAKER_1: I like the book [inaudible 01:11:02].
1:11:02
SPEAKER_1: So the ideas thing.
1:11:04
SPEAKER_1: I think the author is a very interesting character.
1:11:06
SPEAKER_1: I just read several of his books.
1:11:08
SPEAKER_1: Actually, I really like them.
1:11:09
SPEAKER_1: Yeah, I like that book.
1:11:10
SPEAKER_1: Yeah, I think there are a lot of books that I like.
1:11:20
SPEAKER_0: No, thank you for the recommendations.
1:11:23
SPEAKER_0: So thank you for being on the podcast.
1:11:26
I mean,
1:11:26
AI Engineering is such a new field and it was great to hear from yourself who
1:11:30
has clearly gone very broad and also very deep and has been in this field even
1:11:33
before it was called AI Engineering.
1:11:35
SPEAKER_0: So thank you for this.
1:11:39
SPEAKER_1: Thank you so much for letting me ramble on on the show.
1:11:43
SPEAKER_1: Yeah, I really appreciate it.
1:11:44
And I think that one thing for me I really enjoy from writing or talking is that
1:11:49
I get feedback because somebody was like,
1:11:53
"Oh,
1:11:53
I think I'm less interested.
1:11:54
SPEAKER_1: I agree with you."
1:11:56
I mean it's great to hear,
1:11:59
but it can also a little bit not good for the ego,
1:12:01
but I also really like maybe a little pushback like,
1:12:04
"Okay,
1:12:04
you didn't think about this,
1:12:05
you forgot this,
1:12:06
orIdidn'ttakeintoaccountthisone."
1:12:07
SPEAKER_1: So I would love to get those feedback.
1:12:09
So if there are anything that you felt like I missed out on,
1:12:13
do let me know.
1:12:14
SPEAKER_1: Really appreciate it.
1:12:15
SPEAKER_0: Thank you to Chip for this conversation about AI Engineering.
1:12:17
To get in touch with Chip,
1:12:19
including to get feedback on her book,
1:12:20
you can find her contact details on her website linked in the show notes below.
1:12:24
I've also found her book AI Engineering to be a broad and deep overview of this
1:12:30
important field.
1:12:31
SPEAKER_0: It's a book that focuses on the fundamentals that are unlikely to change.
1:12:35
SPEAKER_0: So if you want to learn these, it's a good book to have.
1:12:39
In the Pragmatic Engineer,
1:12:40
we've previously done several AI and ML related deep dives.
1:12:44
SPEAKER_0: Check them out, also linked in the show notes below.
1:12:46
If you enjoyed this podcast,
1:12:47
please do subscribe on your favorite podcast platform and on YouTube.
1:12:50
SPEAKER_0: Thanks and see you in the next one.