Latest / Agentic Conversations (formally mlops.community) / Building Defensible AI Apps // Gregory Kamradt // #191

Transcript

Building Defensible AI Apps // Gregory Kamradt // #191 · Agentic Conversations (formally mlops.community)

0:00
My name is Greg Kamrad. I'm an AI educator and I run
0:03
data independent, which helps developers learn how to build
0:05
their first AI applications. And for my coffee, I just do
0:09
black coffee. I'm going straight.
0:12
Welcome back to the MLS Community Podcast everyone.
0:15
I am your host Dimitrios and today what a conversation with
0:21
Greg, Mr Data Indeed himself. You may know him from his
0:26
YouTube videos that have helped and inspired thousands of
0:30
people. Today I had a blast talking
0:33
about Opening Eyes Dev Day. We brought our own special spin
0:37
and thoughts through the table. Hopefully it's something new
0:40
that you haven't seen from the thousands.
0:42
Of different recaps that are out there on the Internet, we gave
0:47
our two cents on what was special and what we thought
0:51
wasn't so special. And here's a little funny take
0:53
away from that. I thought this AI shoe thing was
0:57
revolutionary. Greg was like, I think it's just
0:59
marketing man. So that was a cool 1 to get into
1:03
debate a little bit. We also talked about the context
1:07
window and how he created a whole special test on where and
1:16
how valuable each part of the context window is.
1:20
He gets into how that works and you can check out a link to his
1:23
whole video that he created in the description.
1:26
We talked about. How he produces content, How he
1:31
goes over different pieces of products.
1:35
Because I love his product mindset and what he brings to
1:38
the table when it comes to creating videos and creating
1:43
products that he then makes videos out of.
1:47
For those of you that do not know, Greg advises startups from
1:51
seed to growth stages and his whole mission.
1:55
His objective we could say. Is to cultivate not just
2:00
awareness, but practical understanding of how to
2:04
optimally utilise a company's tools.
2:09
So I hope you enjoy this one. I was recording it in Portugal,
2:14
having a blast on the coastline right outside of Lisbon when we
2:18
did this one. And one thing that I actually I
2:22
got to say, I love Lisbon, I love Portugal just in general,
2:26
but I forgot how much I love the.
2:29
Portuguese language One thing that those of you who speak
2:33
Portuguese will know, and when you want to say that something
2:36
is going to happen, like it's it's a done deal, you say
2:41
Combinado and it's like it's combined and for some reason
2:45
that just sticks with me. I have a great time when I'm
2:49
there chatting with all my Portuguese friends and listening
2:52
to that beautiful language. So without further ado, let's
2:55
get into it, but not before a quick.
2:59
Word from our sponsors. Ever grappled with the heavy
3:03
lifting of multimillion or even billions scale vector data sets
3:08
in your AI projects? Well then meet Bill.
3:11
This, the open source vector database that's not only highly
3:14
scalable, but it's also blazing fast.
3:17
It's so hot. Created by Zillas in 2017,
3:21
Milvus can effortlessly store, index and manage over a billion
3:26
embedding vectors. It's battle tested by thousands
3:30
of enterprise users. Middle This is built to power
3:34
vector similarity search for retrieval, augmented generation,
3:38
product recommender systems, anomaly detection, image
3:43
similarity applications and so much more.
3:46
With 9 specialised index types and GPU acceleration support,
3:52
Milvus delivers 10X the throughput of other solutions.
3:56
Ready to find out why Melvis is the most widely adopted open
3:59
source vector database? Visit milvus dot IO.
4:04
Or get started in seconds with the fully managed free version
4:07
of Milvus at zillis.com. That's ILI z.com.
4:13
Check it out. All the links are into the
4:14
script. Let's go. here with Mr Data
4:24
Independent himself, Greg, it is great to have you here, man.
4:28
Yeah, absolutely. Thanks for having me, Chris.
4:32
It's perfect timing too, and I feel like the stars aligned
4:36
because last week we were supposed to do this, but last
4:39
week? The whole AI Open AI Dev day
4:43
didn't happen yet, so we would have had a conversation.
4:47
It would have been amazing. I know it would have been, but
4:49
today it's just like even more topical and more perfect.
4:54
Because it just happened. Yeah, absolutely.
4:58
You know, I hosted a watch party here in San Francisco, and we
5:01
had about 100 folks come in and just nerd out on AI stuff,
5:05
right? So obviously there's huge demand
5:07
to get into the event, and the event looked freaking awesome.
5:10
I mean, it wasn't too corporate. It was beautiful, wasn't too
5:13
much hype. I mean, there was a lot of hype,
5:15
but like, I think it was justified.
5:16
But for all the folks that didn't get in, we still want a
5:18
place to nerd out on all this stuff.
5:20
And so we partnered up with the VC, the GP, and that was over in
5:24
Jackson Square in San Francisco, and we just had a freaking great
5:27
time watching it. Nice.
5:29
Yeah. Shout out to Finn and everybody
5:32
at the GP. That's an awesome crew over
5:34
there. And that sounds like the most
5:37
San Francisco thing you could do.
5:40
Have a watch party. Pretty over the line, Debbie.
5:44
Yeah. Well, you know, we weren't the
5:45
only one either. So there is a couple there's
5:47
ever going on. There's plenty of demand for it.
5:49
And it was a Monday at 10 AM. People were working.
5:53
It was funny because the pictures seemed like the event
5:58
was way smaller than I was expecting it to be the open a
6:02
the live dev day. Yeah, the live, live.
6:05
Yeah. I mean, I tell you what the the
6:08
main event looked pretty packed. You couldn't.
6:10
I couldn't really tell how many were in there, but then all the
6:12
different workshops and breakouts that they had, two
6:14
seemed really, really cool. I love seeing the screenshots of
6:17
the slides that we're getting poked around.
6:19
So we should probably just get into it, man.
6:21
I mean, you can't do it. You mean creating YouTube
6:24
videos? You've been doing all kinds of
6:27
incredible stuff around not only software engineering, but data,
6:31
your backgrounds in data. And so the whole LLM revolution
6:36
caught you in a great place. And the YouTube videos that you
6:39
created have helped so many of us, myself included, understand
6:44
what is going on in this space and how to best utilise it.
6:48
And so you have this unique. Viewpoint because I feel like
6:52
you are knee deep in everything that is happening and like I
6:57
consider myself on the cutting edge, but I get the feeling you
7:01
are even more cutting edge than me watching this stuff as it
7:05
happens at the BBC for office. Ohh, So what were some like?
7:13
You wrote the Twitter thread just now, right?
7:15
So there's a lot of things that are probably on your mind.
7:18
I have some things that are on my mind.
7:21
What? What are things that jumped out
7:23
at you right away? Sure.
7:26
You know, I was kind of, I kind of was mentally bucking them,
7:29
bucketing them into different categories.
7:31
The first one I thought was like kind of like the table stakes,
7:34
the standard stuff. Sure, GBT 4 is going to be
7:36
faster, makes sense. GBT 4 is going to be cheaper,
7:41
makes sense. The context link was going to
7:44
increase. That was pretty cool.
7:45
I wasn't expecting that to go all the way up to 128 K, but I
7:49
guess if you have anthropic marketing 100K then you're
7:52
definitely going to have to one of them a little bit anyway.
7:57
So that was all the standard stuff.
7:58
I wasn't too surprised by any of that.
8:00
I'm actually doing an analysis that's pressure testing the
8:02
128,000 context token limit and I'm excited to put that out
8:07
later. But then there's kind of the
8:09
wild card stuff and so the things that really interest me
8:12
were the new modalities, so the text to speech API, cool net,
8:17
new capability. That one is probably my least
8:20
favourite because we already had eleven Labs and Play HD, so this
8:22
stuff was already out there. But if you can consolidate
8:25
vendors and have the open AI product quality, sure, that's
8:29
pretty easy. I like consolidating APIs.
8:33
And then of course there's Dolly 3, which is net new via the API.
8:37
It's like, that's really cool. That's a completely new thing.
8:41
Now, I still think that it's kind of early because we don't
8:44
have as much control over the generated images as we may want.
8:47
The spelling isn't right. You can't go edit the thing
8:50
that's in the top right corner and kind of isolate it as as
8:52
well as I may want want it to be.
8:54
So that one will grow. It's OK.
8:56
And and then before I get to the before I get to the last one,
9:00
The things that I think were a really, really hyped that I'm
9:04
still a little quiet. I'm still questioning is the GBT
9:08
assistance O huge promise, huge hype.
9:12
I think there's a lot of really cool things that are going on
9:14
with it, but after the plugin adoption, I'm not 100%
9:22
confidence. It's going to be a home run,
9:23
right? The App Store.
9:25
I I think it could be amazing. I just I need to see a little
9:28
bit of proof in the pudding and see some user adoption on that
9:31
before I go and I say that this is really freaking.
9:33
Cool and exciting assistance real fast for those who didn't
9:37
catch them. Yeah.
9:38
So open a I they came out with kind of two things.
9:41
One is this concept of a what they call a GPT.
9:44
And so the names of the little controversial because GPT means
9:46
one thing, but they're using it to mean sort of like a bot.
9:50
You can think of it as a prepackaged chat bot.
9:52
So all of a sudden when you create a GPT, you can give it
9:56
documents background context. So everybody and their mothers,
10:00
their first kind of Hello World application they would build
10:02
with language models is they'd create a chatbot and they'd go
10:06
attach internal cut company documents or they'd go attach a
10:10
website so they can ask questions to a website.
10:12
Or everybody. Or I do a lot to the Paul Graham
10:15
essays right? Well, now instead of doing that
10:17
manually, you can go and do that via no code on the UI.
10:22
So you can go create a GPT, which is just like a bot.
10:25
You can attach some files to it, so all of a sudden it can go and
10:28
do retrieval. So you're going to give it extra
10:30
context, which is pretty cool. But then not only that, you can
10:33
also connect different tools to your GPT as well.
10:36
And the big one that everybody loves right now is the code
10:40
interpreter. And so the code interpreter from
10:42
Open AI, that's when they're going to run a little bit of
10:44
code on their end, to answer your question.
10:48
So for example, if you gave them a complicated math equation,
10:51
well, having chat, GPG ChatGPT try to do that in its head, Even
10:56
if you said think out loud could be pretty tough.
10:59
It'd be a lot easier if Chad GBT could write the code according
11:02
to the math equation, run the code and then get you the
11:05
answer. And that's what code interpreter
11:07
does, which is really cool. OK, so then if that's that's an
11:10
assistant, it's kind of like an isolated single bot.
11:13
Well, the next natural thing is let others let people create
11:19
bots and then go share them with other people.
11:21
So they created a marketplace where you can make, you can make
11:25
a bot and then share it with friends.
11:27
And so of course everybody's drawing parallels to the Apple
11:30
App Store. And so you can see like you have
11:33
a bot logo, you have a bot creator, you have a terms of
11:35
service and things like that on these bots.
11:37
And it's like, man, if that takes off, that's pretty cool
11:40
because now you have these specialised tools that you can
11:43
use for different use cases around your workflows.
11:47
And if that wasn't enough, the part that makes it extremely
11:51
interesting is they're doing revenue share with the
11:55
developers who create these bots.
11:58
O now all of a sudden you have an incentive to make sure that
12:01
your bot has a value or else it's not going to, you're not
12:03
going to get paid for it, You have an incentive to market your
12:06
bot. So we haven't seen any ads yet
12:08
for like GPT, but the minute you have a customer acquisition cost
12:12
and the minute you have LTV, well you're going to see a lot
12:16
more paid ads for these types of bots that drive you to the Open
12:19
AI A store. And and so I'm very enthusiastic
12:22
about it but and this is a net new paradigm in the AI world
12:29
which is untested. I have no idea if users will
12:32
naturally gravitate towards these types of use cases.
12:34
I don't know if it's going to flow, et cetera, et cetera.
12:38
So I'm optimistic, but I'm cautiously optimistic which is
12:41
why I'm not declaring victory on this quite yet.
12:43
And then the last thing was that I was most excited about from
12:46
the announcement was GPT 4 vision via API.
12:50
So a couple weeks ago I did a big fat analysis around how
12:53
people were actually using GPT for vision, but it was all one
12:56
shot examples because you were just doing it through the
12:58
through the UI. You give the UI a photo, it
13:01
gives you a single response back and that was kind of the end of
13:04
the workflow. But now we're starting to see
13:06
different types of As take advantage of the iterative
13:09
nature of going back and forth with the AIO.
13:11
For example, the one that's early on is this dude attaches
13:15
GT4 Vision to his webcam, and I think it must just take a photo
13:19
every five seconds or something like that, and then go process
13:21
it. But he'll show different objects
13:23
in his webcam, and then all of a sudden GT4V will tell him what's
13:26
in the webcam, which is pretty cool.
13:29
Nice. So anyway, that's the one I'm
13:32
most excited about, but there's a lot of really cool stuff that
13:35
was announced there. I have one that a buddy was
13:38
telling me about just last week, too.
13:42
When it comes to the vision and what he's been trying to do with
13:46
Vision, but it hasn't been as easy because you haven't had
13:50
that API accessibility yet until now.
13:54
But there are companies out there, and this may seem crazy
13:58
to us. Technologically advanced species
14:02
on the planet, but there are companies that still use fax
14:06
machines and they've got their fax machine hooked U to OK.
14:11
If I get a fax, it gets routed to an e-mail like Inbox, and
14:17
then from that e-mail like inbox, I take those faxes that
14:21
come through and I have someone on my team.
14:26
Take what it is from the facts and usually it's orders that are
14:30
coming through. So they get their orders through
14:32
via fax and it has all the fields in the fax and it's like
14:36
in a PDF and and someone from the team has to transfer that
14:42
data from the PDF to their order form.
14:46
And all of a sudden now this guy was like, you know what we can
14:49
do? We can just throw that PDF into
14:52
ChatGPT and spit out a CSV file. So at the end of the day, you
14:57
can just batch it. I'll throw all of those PDFs
15:01
into it and then spit out one long CSV.
15:04
And there's your order form now. Yeah, yeah, I absolutely love
15:08
it. And we don't yet have access to
15:11
the GT4, the fine tuning, and so it's only going to get better
15:16
from here. But I think that's an absolutely
15:17
beautiful use case. Yeah, yeah.
15:19
And and it's one of those ones where you're like, wow, if you
15:22
just understand. What is possible right now?
15:28
You don't have to make those big swings.
15:31
It's just there are people that are doing things that are so
15:35
painful that they do not need to be doing.
15:38
And if you just understand what you can do and it and 99% of the
15:42
time, let's be honest, it's probably like good enough with
15:45
Zapier integration. And it's just if you know how to
15:49
automate it, then you provide so much value to that company.
15:55
But there are these times where you're like, oh.
15:58
Yeah, you could. You could throw this into.
16:02
GPT or whatever your LM of choice is or whatever.
16:08
If it's just a clip and interrogator XYZ, you could
16:11
throw it in there and do some really useful stuff out of it
16:16
and all of a sudden it frees up that person's time to do
16:19
hopefully higher value work. Yeah, yeah, yeah, I totally
16:24
agree. The hard part with all those
16:25
things is usually just finding the problem that's actually
16:28
there and actually validated and then finding somebody to pay for
16:31
that problem. Once you do that, like you said,
16:34
I totally agree the the barrier to building is pretty low right
16:37
now. So it's just about finding the
16:39
problems in the customer. Yeah.
16:41
And it's like go into these different places where people
16:46
are curious about AI, but they don't understand AI and they've
16:51
got their business and they're probably a little bit.
16:55
Like, they're just worried about their business in the
16:57
day-to-day, and so they can't go and explore what is possible
17:01
with AI. But they're like, hi, I'm
17:02
hearing about a I what can we do?
17:05
And you just gotta be like, what are some processes that you're
17:08
doing? Tell me about your business, you
17:09
know? Yeah, totally.
17:11
I feel like that is something that you, you have a nice
17:15
vantage point on because you do understand all these
17:18
possibilities. And then when you talk to people
17:20
that are doing things, you can say, OK, this could totally be
17:24
fixed with software, right? And I'm not going out on a limb
17:28
and saying that software can fix everything, but I do see that
17:31
there are a lot of things out there that now because of the
17:34
capabilities that we have. They can't be updated for this
17:40
new paradigm. We could say, yeah, I mean, I
17:43
totally agree. My, my background was really on
17:45
the B2B products and like operations side of the house.
17:48
So I've done a lot of building my own tools and I've done a lot
17:51
of trying to dig into customers and figure out what product do
17:54
we build, how do we iterate towards PMF and all of those
17:57
hard problems and like gone through the muck to do it.
17:59
And so it's really fascinating for me as a student of building
18:03
products to look at the landscape right now and see all
18:07
the products that are now unlocked and then all the
18:09
problems that those products could be applied to.
18:11
Yeah, yeah, totally. I mean, going back to this, the
18:15
announcements. I really like that you didn't
18:18
declare a victory on the App Store idea yet.
18:24
It feels like that needs to happen.
18:27
I just wonder if. Open AI is going to be the
18:32
winner there, or if it's going to be like a more generalised.
18:36
Version where you have that marketplace, but it's using a
18:40
little bit because I just foresee that.
18:44
You're going to have. Some really cool.
18:48
Whatever you want to call them, GPS or assistance or whatever.
18:53
You're going to have those, but a lot of them are going to be.
18:58
Quite complex pipelines and. I'm not sure that.
19:04
Within. Or using GPT 3 or 4 or whatever,
19:10
you're going to get everything that you want and maybe you will
19:13
be able to you drink the kool-aid and you go, you know
19:15
what, I got fine tuning. I've got GT4.
19:17
It's the best that there is and I'm just going to go for it.
19:22
But you also have to deal with like.
19:24
They kind of take a lot of the pie, right?
19:28
It's like, huh, I think the Commission or the payout is like
19:34
there's a lot left to be desired.
19:36
I don't think they announced all of it, but it's the same as like
19:39
when you're dealing with the Apple App Store, right?
19:42
And you got to pay out 30% because people buy it something
19:45
on the App Store. But the thing with the App Store
19:47
is that everybody's got an iPhone and so you kind of
19:51
constrained to it. I don't see it as like ohh
19:54
everybody's. Got to use GPT or Open AI as of
19:58
yet. Maybe.
20:00
Maybe they will, but what are your thoughts?
20:02
About that, yeah, totally. I think that's a that's a good
20:05
thought because I I was thinking the same thing.
20:06
If you compare it to the Apple App Store, why was the App Store
20:09
so successful? Well, at the beginning it kind
20:12
of wasn't. They need to deport a lot of
20:14
money into it to get people to build to do the curation right.
20:18
But the difference between the Apple App Store and the GPT
20:20
store is the App Store was constrained to the iPhone.
20:25
The iPhone is a piece of hardware.
20:27
Once people bought the phone it was very hard to switch off the
20:30
phone. So you got the apps in the 1st
20:33
place. But then not only that, the apps
20:35
kind of sold the iPhone because Apple would promote the app
20:39
which would then go and promote the phone and then they would
20:41
just help each other out, right? Which is pretty cool.
20:44
So if you were to look at that analogy and compare it to Google
20:48
Anthropic Open AI, you know, Meta and all these groups, you
20:52
know, a couple of weeks ago you could say that the model was the
20:55
equivalent of the phone. But especially with tools like
20:59
lane chain, you can switch models extremely easy.
21:02
You can even host your own freaking model on your local and
21:04
it's only going to get easier to do that.
21:06
OK. So there's not much lock in or
21:08
defensibility on just the model piece other than generalised
21:10
intelligence, cost, latency, etcetera, etcetera.
21:13
But those things you can work around.
21:15
So the part that's interesting about the move with Open AI is
21:18
how they're choosing to build lock in with developers and with
21:22
functionality. So it's not a it's not a mistake
21:27
or it's not a coincidence that they're hosting or files for
21:30
you. Yeah, right.
21:31
It's not a coincidence that they're running code interpreter
21:34
code for you because that's the more you build on top of that is
21:38
the more lock in as a developer you're building.
21:41
That's going to be harder to do for Google.
21:42
And that's what's so fascinating about thinking about this AI
21:45
race is if developers get used to perfect JSON output from Open
21:50
AI, function calling from Open AI, code interpreter from Open
21:53
AI, what is the response going to be from an anthropic in a
21:56
Google? Do they have to build those to
21:58
have table stakes? I don't know.
22:00
Do they go and do something better?
22:02
I I don't. Know, Yeah, it's gonna be weird
22:06
if you can't output Jason all the time, right?
22:11
Like as a developer, you're not going to put up with it because.
22:15
You have been getting it perfectly.
22:18
With this other tool. So it is, yeah.
22:20
It does feel like it quickly becomes table stakes.
22:23
Yeah. Yeah.
22:24
It's funny that you mentioned the model and the lock in.
22:28
They're very they are just eating like the surface area
22:33
that they're eating up. And the way that they're
22:36
growing. And going more and more like
22:40
expanding more and more is. An interesting one to watch
22:45
because it had already felt like, wow, they were a monopoly
22:50
and now it's just like, and now they're eating up.
22:53
They're trying to like grow and grow and grow.
22:55
And so I. I wonder about that because
22:59
there's a lot of surface area that they're trying to cover.
23:02
And is it going to be like the? 1st generation data tooling that
23:10
came out in whatever 20/15/2016 or or even before that like the.
23:16
Domino data Labs and where their platform and they kind of do
23:20
everything, but they don't do things as like machine learning
23:26
moved on. It was like, well, this isn't
23:28
really what I'm looking for in my machine learning playbook
23:32
anymore, right? Yeah, but they captured a lot of
23:35
that initial market that was doing stuff because they were
23:39
the platform in the beginning. So it feels, it feels like a
23:43
little bit of that play. But.
23:46
Considering how much surface area they have and they're
23:50
taking U. You can't do all of it well,
23:54
right? Like or, I mean maybe you can,
23:57
but it just feels like it's a much harder play.
23:59
Yeah. Well, I mean on that note, they
24:01
are definitely increasing their product surface area, that's for
24:04
sure. And you need, if you want to
24:07
keep the quality bar, you need people who can think about how
24:09
to keep that quality bar. I'm pretty sure they've
24:11
increased their head count size. You know, I'd have to look up
24:14
exactly how much they have right now.
24:15
But I mean it's absolutely massive.
24:16
I think there were 300 and now they're like 700.
24:19
They're going quick, yeah. They're grown quick and so it
24:22
looks like this is 2 to 500 employees on LinkedIn still,
24:26
which seems a little small, but maybe not.
24:28
But yeah, and you know, I bet Open Eye wants Google to do
24:31
something. Because Open AI, they want the
24:34
spotlight, but they don't want to be the freaking 98% market
24:37
share type of spotlight. They want Google to come in and
24:41
go do something with them. So I think another interesting
24:44
question, I'd be curious to hear your thoughts on this too.
24:46
You know, part of the, in the back of my mind, I mean, I like,
24:50
I'm fine with Google. Some people hate Google.
24:51
I'm fine with it. I'm rooting for him.
24:53
You know, I'm a capitalist at heart.
24:55
And so I love to see companies compete for value against
24:57
consumers. I thought Google was just slow
25:01
for a little bit and I was like, oh, they'll get the act
25:03
together, they'll come out with something.
25:05
But when you see something like Vday from Open AI and then you
25:09
saw the Suhail's tweet, that was where period is period Google
25:14
period. Yeah.
25:16
And it's. I'm just wondering, is there an
25:20
actual problem that we're not seeing good product and good
25:23
innovation coming out for them or is it just, no, they haven't
25:26
pulled the trigger yet? What do you think?
25:30
I mean, from what I. Heard from friends that work
25:33
there is that they are a little bit more secretive about what
25:39
they're doing right and. I think there's It feels like
25:44
it's twofold. One is that.
25:47
They're trying to build in a I to the products that they
25:50
already have as opposed to go and create an AI product at like
25:55
a new AI product, Sure and. So that that feels like kind of
26:01
weird though because Google, when I was growing up, Google
26:04
was all about the moon shots and you know you had what was it
26:07
like Google X or something? Google what was the moon shot
26:11
one that they had. Yeah.
26:13
And. But yeah, I I heard, and this is
26:19
all hearsay, and I don't know, a friend of mine told me who
26:21
worked there and works there. He was like.
26:26
The vibe that he got with the researchers, the ML researchers
26:30
at Google, is that they were very open when it came to
26:35
putting out all these papers and putting out all of this
26:39
information on this research that they were doing.
26:42
And then they put it out there only for others to capitalise on
26:47
it. So now they're not as interested
26:50
in putting it out there, right? Sure, sure.
26:53
Yeah, I wonder if this it's the researchers who thought that or
26:57
if it's the the suits running Google.
27:00
They think that but. Yeah, probably a little bit of
27:03
both. That's, yeah.
27:04
But either way, I tell you what I'm I'm rooting for them.
27:08
As a consumer, I love companies battling for value and so I hope
27:14
it works whatever it is. Yeah, you know one thing that
27:18
came out of this dev day that I thought was probably for me.
27:22
The most interesting that I didn't see coming at all was the
27:27
shield and being able to protect you against copyright.
27:30
Yeah, yeah, yeah, yeah. You know, I I made my top ten
27:33
list and I actually left that one off.
27:35
The top 10 had a few. Really.
27:39
That was my number one. That was all for your top ten
27:41
completely. Well, I tell you, I tell you
27:44
why, because I mean, it kind of seems like more of a marketing
27:46
play than anything because they need to reinforce the message.
27:50
Or I guess what what I heard from somebody who was familiar
27:53
with it was that from a sales standpoint, companies were
27:57
worried that they would get sued for using Open AI.
28:01
Because yeah, of people who claim copyright material on
28:05
there now just like how Open AI says we do not train on anything
28:10
that comes through the API or ChatGPT Enterprise.
28:13
I think it was just more of a marketing play because like what
28:15
did they actually say? They said like, oh we'll help
28:18
you with legal and pay your costs if it comes in.
28:20
But like it felt a little fuzzy with that.
28:22
Ohh really. But then on the other hand, I
28:25
thought, well, if you're Open AI and you have a legal team of 20
28:30
people whose specialty expertise 100% is to grind on copyright
28:36
law, then then you're basically kind of just like outsourcing
28:39
your legal team on who specialises in copyright law.
28:43
And I don't know if you're gonna win against that type of team,
28:45
if they do it over and over and over again.
28:47
So I think it was more marketing, but who knows?
28:50
Yeah, it's funny to think about. I guess it just is very much
28:55
marketing to make people feel comfortable using.
28:59
The output in their tools right? And allowing whatever comes out
29:06
to be. OK.
29:09
Ahead. Now that you say it like that, I
29:11
can see that point. I can 100% see it's just like,
29:15
yeah, you this class action lawsuit are loyals will handle
29:19
that. You don't have to handle it.
29:21
We'll, we'll take care of it for you.
29:23
Yeah. Yeah, yeah.
29:24
I see that point for sure. I mean one other piece that I
29:28
wanted to talk to you about was when it came to the
29:32
orchestration layer and. Delaying chains in the luma
29:36
index out there. I'm not sure if you saw Jerry.
29:40
Community member. Creator of Llama Index.
29:44
He threw out that tweet like people are asking if we still
29:48
need a tool like Lama Index. Now that Open AI released
29:51
everything and I thought it was quite thoughtful how we put it
29:55
out, but it is. You could feel that there's
29:57
like. A bit of existential crisis like
30:02
Ohh will, I guess. If Open AI does this all for us,
30:06
we no longer need Lum index to do any of it for us, right?
30:11
How do you feel about that whole situation?
30:13
Yeah, you know first thing I'll say is when plugins came out
30:16
people were flooding my DMS with Is lane chain dead?
30:19
It's done alright since then. Can you boom.
30:24
Since then I I agree with Jerry and so anybody who's worried
30:29
about it, I would turn that person and say OK, so you want
30:32
open the Ido retrieval on your docs.
30:34
Ohh cool. What's the retrieval method?
30:37
They're doing and as far as I've seen, I've seen, I've seen
30:41
speculation. I haven't seen any like concrete
30:43
facts about how they're actually doing retrieval.
30:45
And then I'd say ohh cool, is that method optimised for your
30:48
vertical and your use case? Oh well, if you don't know, well
30:52
wouldn't you want more control over it?
30:54
So that's that's my kind of sassy way of saying that I my
30:57
expectation is that Debs will want more control over the
31:00
retrieval process than Open AI will give them.
31:02
And while they probably have a really good retrieval, they have
31:05
to have a generalised retrieval service, which they have to,
31:08
they have to try to make everybody happy, which is really
31:10
tough. Yeah.
31:12
Hey everyone. My name is Aparna, founder of
31:14
Verizon, and the best way to stay up to date with Emma Lops
31:17
is by subscribing to this podcast.
31:21
O. My hypothesis is that is that
31:24
for no code builders, for people who are going to the UI and like
31:27
building these GTS, Sure, yeah, throw throw some of your PDFs in
31:30
there, throw your employee handbook, all that.
31:33
I'm sure it'll be fine for those use cases, but when you're
31:36
building a specialised tool, I don't think that you're going to
31:40
want to rely on Open AI to control the quality of your
31:44
retrieval because there's just too much to tune for your
31:47
specific use cases. O I'm in Garry's camp, I believe
31:50
it. I think that the lane chains and
31:51
law indexes retrieval wise will still be there.
31:56
Have you played around with this one?
31:58
I just know because I interviewed the creator and this
32:02
is a bit of a tangent, so forgive me, but yeah.
32:04
Yes, π. Have you played around with yes,
32:07
pie at all? I've heard of it.
32:09
No. Dude, I think you're going to
32:11
like it. So that's just a little bit like
32:14
put a fork in that one, go check it out, have some fun, make some
32:18
YouTube videos with it. The idea of compilers just had
32:22
me. Literally, like I'm still
32:25
thinking about it and I talked to the creator I think 3 weeks
32:30
ago now. So yeah, I'm I'm a huge fan.
32:34
It's very much in that orchestration layer and playing
32:37
around, but the abstraction is quite nice and the compilers are
32:43
something that I hadn't thought about and it reminds me of.
32:48
That I've been watching a little bit of that series.
32:50
Loki, I don't know if you know that series.
32:55
So they have this like one timeline which is like the
32:58
sacred timeline and then there's so there's like life that is
33:02
being lived on that timeline. But then you you have like these
33:05
infinite other number of timelines that are branching out
33:08
of this one timeline. And so the compiler in my mind I
33:12
I like envision it like this because they're trying to do it
33:16
says basically. It's like, hey, there's there's
33:21
this one way to do this thing, but let's test all these other
33:25
ways to do this thing and whichever one we find is the
33:29
optimal solution. So maybe that comes with
33:33
prompts, or maybe that comes with like you gave us this
33:38
prompt or these different functions and we're going to see
33:43
if we can. Find better ways for this to get
33:48
the output that you're looking for.
33:49
Yeah, I know it's one way of doing it.
33:51
I'm probably like. Chopping it UA little bit and I
33:54
may not be. And like I said, it was a few
33:56
weeks ago that I talked to him. But I walked away from that
33:59
conversation. Blown away.
34:01
Yeah, that's absolutely cool. You know, it reminds me of I'm a
34:04
big David Deutsch fan. And he says the way that
34:06
progress gets well gets forward is you make your hypothesis and
34:11
then you put it up to criticism. And if it can go past that
34:14
criticism, then you're good to go.
34:16
So, much like you're saying, you make all your different
34:18
hypotheses which are trials as to what you think would improve
34:21
your thing. Put it up against the test which
34:23
is evals the criticism and see which one pops out the other
34:25
end. Yeah, yeah, yeah, dude, so.
34:29
Yeah, check it out. I mean, back to this, the
34:32
orchestration layer not being dead.
34:35
Yeah, And this whole idea, I think the other thing that Jerry
34:39
said that I'm 100% on board with and you can see the clear trend
34:43
happening. You mentioned it earlier.
34:46
Models are getting smaller and it's getting easier to run them
34:49
locally or at least like semi locally and.
34:55
I get the feeling that that's going to be a huge value add
34:59
because now you have control of the model, right?
35:03
It's like full circle. Like we went really big.
35:07
First we started small, then we went as big as we could go, and
35:10
now we're like, wait, actually, it might be better small.
35:13
Yeah, Yeah, totally. You know, I think I measure it
35:17
in terms of this is kind of like the corporate way to say, but
35:22
like new workflows that are unlocked.
35:24
O what I mean by that is could I have a language model on my
35:27
iPhone right now? Sure, Yeah, totally.
35:29
It'd be really small. It wouldn't really do much
35:30
either because it couldn't do anything.
35:32
So it's not very useful to me, but I think one of the
35:34
milestones on a B2C level is at what level can you have an
35:40
actual conversation with a language model on your local
35:43
phone? Well, you need the latency to go
35:45
down. You need a pretty good model.
35:47
So it has some personality that comes along with it, right?
35:51
That feels like a really good milestone for that.
35:52
But then OK, that milestone kind of stays locked in place.
35:56
You may want to ask more questions, etcetera, etcetera.
35:58
But the problem or the wrinkle with all this is yes, models
36:06
will get local, but my hypothesis is that we'll never
36:10
be done with getting better and better models on our phone, just
36:12
like how operating systems get better.
36:14
They've been getting better since the first iPhone.
36:16
Well, why do they need to get better?
36:18
Well, we have more photos. We have apps that require more
36:22
compute, et cetera, et cetera, et cetera.
36:24
So we used to be able to take a take a photo on our phones you
36:28
know 15 years ago that functionality still stays the
36:32
same, it just got a little bit better.
36:33
That's one of those milestones. But now we have more complex
36:36
apps and all this other stuff and so that's kind of a long
36:39
winded way of saying we will get local models, but just how OS is
36:43
progressively get better and better, we will never be done
36:45
and we'll always need new models.
36:48
Yeah, they'll continually get better and better.
36:51
And then the real interesting piece is like how does your
36:56
personalised model fit into that or how does is 1 model or two
37:02
models like? I have no idea what the future
37:05
looks like with. These models or what the use
37:09
cases are, and these workflows that you're unlocking, as you
37:12
said, because it's like for me, I'm trying to figure out what
37:16
exactly does it mean. But I'm also trying to keep my
37:19
feet on the ground and not be like, hey, I was going to change
37:21
everything. Yeah, I totally agree.
37:23
And, you know, it's funny. You bring up the one model, 2
37:25
models, three models or whatever.
37:26
Let's use the GPT as an example. Let's just make the assumption
37:30
that they're all powered off of GPT for the new preview version.
37:34
Well, that's just one model. So what's different about them?
37:36
Well, how they're configured is different O the files that they
37:39
have, the prompt that they have, the tools that they have access
37:41
to. And so while it may seem like
37:43
multiple personalities and multiple models not, it's just
37:46
one model with a bunch of different configs, it's stacked
37:49
on top of it. So I wouldn't imagine, I
37:51
wouldn't be surprised if that's the paradigm that we see as
37:54
well. It could just be low number of
37:56
models, you really don't need that many.
37:57
You just need the config on top of it to change.
37:59
Yeah, so how do you look at? Because you've seen all kinds of
38:04
cool apps and I think I feel like you play a lot in the
38:08
application. Layer, right.
38:09
Yeah. That's why a lot of times on
38:11
this podcast, we'll talk about the infrastructure and we'll
38:14
talk about even like a little bit deeper, like what is Open AI
38:18
using to create the GPT, right. But then you can go up the stack
38:23
a little bit and say, OK, now that we've got these LMS. And
38:29
we've got the stack that we're using these LMS for.
38:31
There's an application that comes and serves the end user.
38:36
And so I feel like you've got a great lay of the land of that
38:40
landscape. And also.
38:43
And understanding of like how to make an application that is
38:48
built with AI defensible if we are.
38:52
In one way or another using the same.
38:57
Models using the same orchestration frameworks using.
39:01
We've all got access to kind of the same tools.
39:04
So then it's like the special sauce is is what?
39:09
And I understand that an easy answer for that is like, Oh
39:12
well, the special sauce there is the prompts.
39:15
But. That just doesn't feel like it's
39:19
defensible, right? Like, and especially with.
39:22
Once you dig into DSI, I think you'll see why.
39:25
One thing that Omar, the creator of that, was talking about is
39:28
how. Wrong.
39:30
We are to assume like the best way of creating apps is like
39:36
prompting first. And.
39:39
So anyway, that's just a long winded question on like how can
39:43
we look at defensibility? Yeah, totally.
39:47
You know, to answer your first question very briefly, like it's
39:52
all about finding customer problems.
39:53
Like I said earlier, in the pod you've got to find customer
39:55
problems and you got to find people to go pay for those
39:58
problems. If you can't do that, then it's
40:01
going to be really tough to build out your in the 1st place.
40:03
And builders have a romantic idea that if you build it, they
40:06
will come. But like, there's no one way to
40:11
get success. The other quote that I love, I
40:15
heard on my First Million was that if you follow Warren
40:18
Buffett's rules, you'll never be Peter Thiel.
40:21
If you follow Peter Teels rules you'll never be Warren Buffett
40:25
right. And So what I'm the the the
40:27
reason why I say that is because as advice is grayscale and go to
40:31
market as grayscale. So there's a million ways to
40:33
create products, but the common thread against products that are
40:38
successful that I've seen is they all start with a customer
40:40
problem. It's like the Y Combinator axiom
40:43
makes something people want, and how.
40:45
Paul Graham also emphasises make a small group of people love
40:49
your product, then a big group of people just like your
40:52
product, right? O that's that's number one.
40:55
Then when we talk about defensibility, the that essay
40:59
that I always reference, and I really try to read it like
41:02
almost every six months at this point, is Elad Gill's Essay on
41:06
Defensibility. And the TLDR of it is for the
41:11
the the things that make an AI app defensible are the same
41:14
things that make a foundational traditional business defensible,
41:19
right. And so you're going to have your
41:22
network effects. Can you build those in your
41:24
platform effects, your sales and your distribution, your
41:27
partnerships, your brand? You know, all of that.
41:30
Now it's funny that you bring up the prompt thing because it's
41:33
like, ohh, our prompts defensible.
41:35
Well, that's just one way. That's just a form of tech
41:38
intellectual property, right. And so how long will it take for
41:42
somebody to figure out your prompts?
41:44
Well, maybe not too long. All right, So that's not too
41:46
defensible. You may have like a couple of
41:48
weeks and start on people for that.
41:50
So the only difference in the AI world is that the barrier to the
41:54
tech aspect of defensibility is dropping.
41:57
And so you want, you really want to look towards other mediums of
42:01
your defensibility in order to really build it up from there.
42:04
And, you know, a very correlated question that's super hot right
42:07
now is what's everybody's opinion on ChatGPT rappers.
42:11
Oh, it's just a rapper, you know, it doesn't do much.
42:13
It's not defensible. It's a bad business.
42:16
My view on it is, well, customer energies, customer energy.
42:20
Customer energy is one of the hardest things to get when
42:23
building a product. If you build a wrapper that's a
42:26
simple, simple, simple wrapper and you get 1,000,000 users
42:28
overnight. I don't.
42:30
I don't care at all. I think, holy cow you freaking,
42:34
you have lightning in a bottle. Now go do your work as a good
42:38
business person to go capture that and go make it more
42:41
defensible. But if you told me do I want a
42:43
defensible business with no users or do I want an
42:46
indefensible business with a million users that I can go
42:48
iterate on? I'm picking the ladder all day
42:50
long for sure. That is a great answer and that
42:53
it makes me want to go in so many directions, but I man.
43:00
I love this idea of how, yeah, you got to figure it out first.
43:05
You got to figure out what people want and what they're
43:08
actually willing to pay for, right?
43:10
And then if it's not defensible, it reminds me a little bit of
43:14
the craze with the fine tuning on the image generation.
43:21
Like when stable diffusion came out then all of a sudden you had
43:23
everyone able to make their own selfie and some people were
43:28
going and they were fine tuning. Zero the stable diffusion model.
43:32
Others were going and justice using lensa or like photo
43:37
profile, AI photo, whatever. Ick.
43:40
And you had twenty of those come out and I heard a stat at one
43:45
point that like Lenza was making. 6 billion, they made 6
43:51
million in profit or something like that.
43:54
So the thought that I was that I had was, and what I've talked to
43:58
people about is how, yeah, they're probably.
44:02
Are a lot of really obvious use cases.
44:07
And those are the ones that you are going to have a harder time
44:11
defending against, right? So like.
44:14
A photo generator. A selfie photo generator is a
44:18
quite obvious use case. And then?
44:22
But then there's. There's use cases where you go
44:25
deep into a certain field and you recognise that you can use
44:30
AI for this. Like maybe it is.
44:32
A dentist that now can create the ideal looking facial or
44:38
teeth structure because the stable diffusion is fine-tuned
44:42
on like perfect teeth. I don't know.
44:44
I'm making shit up now, but. What I'm what I'm getting at is
44:49
that there is like this long tail of use cases that are so
44:54
niche. That you have open area, you
44:59
it's totally Greenfield, right? Yeah, I think you're completely
45:03
right. I think that there's power in
45:05
niching down because then not only is the opportunity less
45:09
attractive for big businesses because the the team isn't there
45:12
for them, but the opportunity is still attractive to indie
45:15
builders and small businesses because there's enough for
45:18
there. I mean, is Google going to go
45:19
after an opportunity that's, you know, 1,000,000 bucks a year?
45:22
I mean, probably not. It's not high on the priority
45:24
list, but isn't indie dev going to go after that?
45:27
Heck yeah, they will. And so I think that niching down
45:30
is completely a wonderful idea. I love it and I'm sure Paul
45:34
Graham has some good quotes around niching down, but they're
45:36
not coming to mind right now. Yeah, that's that's the Ohh man
45:43
the. That's great how you reference
45:46
it. You've been playing around with
45:48
Paul Graham as you're like Lorem Epsom for so many different
45:52
tools that you build that you, you just have them ingrained in
45:56
your head. All that was essay #5.
45:58
I remember that one. It's true.
46:00
I mean, it's there, yeah, that's what I use for like a lot of my
46:04
like background contacts and so like I was like essays memorised
46:07
in my head and which ones are short one and which ones his
46:09
longest one. You know, all that stuff.
46:13
Again, like that is the most San Francisco thing that you could
46:18
be telling me right now on this call.
46:21
It is so good. I was at my high school reunion
46:23
a little while ago and I was talking to this girl and she
46:25
said, what do you do? I said, oh, I work at, I think I
46:29
work at Salesforce. She goes, oh, so you're a tech
46:31
boy. Here's the tech boy.
46:33
And ohh, you know, she said in a derogatory term, whatever.
46:38
But you know, it's really good if you can have self-awareness
46:41
and you know yourself. And you know what, Demetrius?
46:43
I'm a tech boy and I don't give a shit because I like it.
46:46
It's what I like. It's what I'm passionate about.
46:48
Own it. I like doing it.
46:49
So I'm a I'm a freaking own it. Exactly, man.
46:54
There's no, there's nothing wrong with that.
46:56
No, no. No, you have been making some
46:59
really cool. Videos when it comes through and
47:03
it's it's not only around LLM MMS. And you're bringing
47:08
different modalities into the mix, right, so.
47:12
You think? Yeah, I I it's almost like
47:16
three-dimensional where you're like, OK, well we have in image
47:19
generators can do this, LM can do this and then we have audio
47:24
to text and then we have text to audio.
47:26
So umm, and you're able to come back to thinking about product
47:31
in that way. And there's one that I know that
47:34
we had talked about for I think you were you were doing
47:38
something where it was with startup pitches where you record
47:41
the startup pitches. Yeah.
47:42
And then you get a bunch of information.
47:44
Can you explain that? Because I thought that was
47:46
really innovative and the way that you come up with these
47:50
different ideas is super cool because you're doing it.
47:54
I mean, you have these products, but you're not really taking
47:57
them to market. You're just showing other people
47:59
how to do this and learn from it, right?
48:02
Totally. So I was really into the idea of
48:07
using a language model but for a live event, right?
48:10
And so I was, I was really trying to think about where
48:12
could I bring this for a live event.
48:13
And it was kind of like the wearable trend that just came
48:16
out with a rewind and tab where you're going to wear the pendant
48:19
around your neck and it's going to like listen and stuff like
48:21
that. I thought, you know, I want to
48:22
do one of those, but I want to do it for a niche topic.
48:25
So I don't need it to follow me around my entire life, just this
48:27
one specific instance. And I was thinking about where
48:32
am I going to do is, I'm going to do this.
48:33
And then I had the idea, oh, that'd be kind of cool if I did
48:36
it as if I was an investor listening to a startup's pitch
48:40
O. What I would do is I would
48:41
record the startups pitch on my phone and click record or
48:44
whatever. It would have the audio get the
48:46
transcription using one of my favourite transcription tools
48:50
diagram. And then I would take that, give
48:55
it to a language model and then step one is just like, oh,
48:59
analyse it and summarise it for me, OK, that's all the easy
49:01
stuff. But Step 2 is like, Oh well no,
49:04
go connect this to the outside world.
49:05
So go get news about the industry that that startup is in
49:08
that may be relevant to what the pitch is telling you about.
49:12
Give me follow up questions that would poke holes within the
49:15
startups page that may be pertinent for me to ask about.
49:18
And so then all of a sudden I'm like, holy cow now I could have
49:21
an intelligent companion for me for startup pitches and I could
49:25
questions I would have never like remembered off the top of
49:27
my head. And it might have been rude to
49:29
go, well that you'll look up or whatever.
49:31
And so I'm currently in the process of building that.
49:34
And believe it or not the biggest blocker for that is I
49:37
can't find many startup pitches out there that aren't Web Three.
49:40
And I don't, I don't really want to go to a web three startup
49:43
pitch. So I'm I'm looking for more
49:46
events to try to go to and test this out on.
49:49
This, that's where I have like a pattern interrupt because I'm
49:53
like the amount of companies I feel like that are raising in
49:57
the AI space. I've thought it would be like
49:59
shooting fish out of a barrel, man.
50:01
I think those are all happening behind doors and like NBC's
50:04
offices. I haven't seen a lot of events
50:06
where I need the one where there's an intimate setting, the
50:09
startup gives its pitch and the audience can ask some questions.
50:12
And so I'm still in the hunt for that one, yeah.
50:15
Yeah, that sounds awesome. Yeah, all these A I companies
50:19
are just getting funded after the first VC meeting.
50:22
Exactly. They're just going here.
50:25
What, what was the valuation? 100 million.
50:27
Alright, No product, no revenue, no traction.
50:30
Take my money. Well, you know, it's funny
50:33
because when I realised that it was really in the smart
50:36
companion app for a live event, I thought, Oh well, actually
50:40
there's a bunch of these. What about a lecture at your at
50:43
your university? That's pretty cool to have.
50:45
Like a little painting that's like live annotating the lecture
50:48
and giving you supplemental information.
50:49
OK. And of course, you know, there's
50:51
that tweet that went viral around the guy who is using it
50:53
for a job interview. You know, like the interviewer
50:57
would ask a question and then the GBT would tell him the
51:00
answers on what to ask where. It's like, OK, that's kind of
51:02
interesting. And like this really kind of a
51:05
lot. Like this would never work.
51:07
But like, there's a lot of first dates that people go on and a
51:10
lot of dates they don't. They need to.
51:11
Maybe they need a companion on like things to ask and how to
51:14
go. Well, on that, there's the
51:15
school aspect. There's the startup itches,
51:17
There's sales for sure. In person, sales is huge.
51:21
There's conference, you know, you're at a booth and you're
51:24
talking to a customer, you know, just a bajillion things where it
51:26
helped to have a live companion. So I think that's kind of cool.
51:30
So now you realise that it's like take me through the whole
51:35
idea from idea to then productizing it to then ohh
51:40
actually this is bigger than just.
51:44
A startup itch companion. Yeah, um, the usual.
51:50
The place where usually starts is I get the original idea,
51:53
which again, we talk about going niche while startup pitches
51:56
niche. OK, cool.
51:57
And then I just I was building an MVP.
52:00
So go record the audio. OK, now take that audio, get the
52:03
transcription, get the transcription, go pass it to the
52:05
language model. And then it's like, ohh, well,
52:07
at the language model, you need questions, You need news, you
52:09
need follow-ups, you need go can you can you find out who the
52:12
founders are and what their backgrounds are so you have it
52:14
immediately on there, you know? And then it's like, wait, wait a
52:19
minute, This is kind of cool. Let me put this out there and
52:22
let me cause. Luckily, one of the nice things
52:24
about having a little bit of a Twitter following is you can
52:26
pressure test ideas pretty easily.
52:27
You just throw it out there and see what the reaction is.
52:31
If that reaction motivates you enough, cool.
52:34
Well how would you generalise this?
52:38
Because really, like you said, at the end of the day it's a
52:39
bunch of prompts, it's a bunch of retrieval, and so it's not
52:43
that hard to generalise these things, you know?
52:47
Especially I wonder, I didn't explore it too much, but even
52:49
with the GT S that are coming around, they have access to the
52:52
web. Can you use them to go do a lot
52:54
of this stuff? Who knows?
52:56
Have you thought about? Are you going to try and create
52:58
a GBT? Ohh yeah, for sure.
53:01
I'll, I mean, I'll mess around with it.
53:02
I'll play with it. You know, if GPT's work, there's
53:07
a paradigm for where you run an agency that only creates and
53:10
maintains GBTS. Sure, like that's a playbook
53:13
that's out there. We've seen what the App Store,
53:15
we've seen it happen. But many times that will happen
53:17
with the BT Store, right? Is that an attractive model?
53:22
I mean, it sounds kind of fun because you're basically like
53:24
GPT Studio and you're just creating these things and
53:26
finding what's going to have value and you know, reaping in
53:29
the the benefits from there from a testing standpoint, 100%.
53:34
I mean as if you're playing in this game, you gotta go try it
53:37
out and you gotta see it. How you think through the
53:41
product is fascinating to me. And then being able to make
53:45
content out of it and recognise like ohh here I should probably
53:50
see am I on track. Let's let's ask the Internet and
53:55
put yourself out there. I think is a huge it's, it's a
53:59
little bit courageous on one hand, and it's also really cool.
54:02
To see and watch how you can come U with these different
54:06
ideas and then you you create them and so I am a. 100% excited
54:15
about that video when it comes out and your other videos.
54:19
I mean, I know you told me a little bit about the video that
54:22
you're creating with the new Open AI updates.
54:24
I don't know if you can tell everyone by the time this.
54:28
This will go out next week I guess this podcast so nice.
54:32
Maybe your video is already out by then.
54:34
Yeah, yeah, it's it's weird. I, I I'm not used to talking in
54:38
the future like that. But you're right.
54:40
So 128,000 contacts token limit. I mean, that sounds pretty cool,
54:44
right? But the number one question that
54:45
everybody has is how is retrieval?
54:47
Is it going to work still? Is it not going to work?
54:50
And so I'm doing what I'm calling a needle in a haystack
54:53
analysis. And what that means is I have
54:55
just a single sentence that is that has a fact in it and the
54:58
fact is just a playful one. It said the best thing to do in
55:01
San Francisco is go to Dolores Park and eat a sandwich on a
55:04
sunny day. I've done it many times.
55:07
Go get an ice sandwich and go sit in the park.
55:08
It's beautiful. And So what I do is, is I stick
55:12
that single fact somewhere in the middle of 120,000 tokens
55:17
worth of context. And then I give that to Chad GBT
55:19
and I say, hey, what's the best thing to do in San Francisco
55:23
based on the context up above? And I see if I can pull it out
55:27
right so that that answer is yes or no.
55:31
And it's not very exciting. So what I wanted to do was I
55:34
wanted to do it at various context lengths, over 1000
55:38
tokens worth of contacts, 10,020 thousand, 30,040 thousand,
55:41
50,000 and do it iteratively across the whole thing all up to
55:44
128 to see to see if it could do it or not.
55:49
OK, cool, that's another good step.
55:51
But being the data person, I have a heat map in my head.
55:54
And so then I thought, well, the other thing that people talk
55:58
about all the time is the position in the document in
56:00
which the fact is held. And so then I was like, alright,
56:03
well let me make this a 2D access.
56:05
And let me actually put the fact that the very top 10% the way
56:09
down 20%, the way down 30 percent 40%, fifty percent 60%,
56:12
etcetera. And let me see how it does with
56:15
retrieval at various context links and various document
56:18
depths and let me go see what it is.
56:21
And I just spent like freaking $200.00 worth of GBT 4 Pro API
56:25
cost to go run this. But they got to be sponsoring
56:28
you by now this is. Crazy.
56:29
This is absolutely nuts. I should have really got a
56:33
sponsor for like 1500 bucks and just went freaking crazy on
56:35
this. But I didn't.
56:36
But anyway, I have the results and they're pretty interesting.
56:40
Actually, I'm not. No, I'm not going to spoil them
56:42
right now. You have to go find me on
56:44
Twitter. Find me on.
56:45
Exactly. We'll drop the link to that
56:47
video in the description for anybody that wants to know.
56:50
My biggest question is, have you also done it with Anthropic?
56:54
That's a great question. No, no, I haven't.
56:57
But I mean, all all these are just parameters and so I could
57:00
just throw it into anthropic and we'll see what it is, yeah.
57:02
Yeah, exactly. And then compare the two and see
57:05
which one is better. One of my pet peeves about this
57:08
entire industry is the lack of digestible benchmarks.
57:12
It just drives me nuts because AI used to be just all
57:15
academics, right? And they're used to these random
57:18
asked papers in black and white that are on archive or whatever.
57:22
Well, with all the business people and all the laymans and
57:24
all the normies like myself that are now in the industry like
57:27
getting further, I need something simple.
57:29
I want freaking model benchmarks COM and go give me exactly what
57:32
I want to see updated daily and I don't have that yet.
57:35
And if a way to monetize that a little bit more, I may go after
57:39
myself, but I don't yet. Is that wait, is that?
57:43
Domain free. I looked it up.
57:45
I wouldn't say that domain unless it was free and it's not.
57:48
Ohh. Man, who's got that domain
57:51
model? benchmarks.com.
57:53
Show yourself and give it to the people so that we can do that.
57:57
That is so good. And I feel for you, man.
58:00
I'm 100% understand what you're saying.
58:03
I would. I almost look at this.
58:07
This idea of being able to grab the.
58:11
The proof and again little tangent but I got I was biting
58:15
my lip not to crack a San Francisco joke in there on it's
58:20
the fact that you're asking for a good point and that is a good
58:24
point. And then I was also you didn't
58:27
tell what the the other text was that wasn't the fact that they
58:31
were looking for. But I mean yeah yeah yeah.
58:33
Well, so where am I going to get 120,000 tokens worth of good
58:36
stuff that's like random? Well, duh programme has 180 S
58:41
and so I used all of his essays as the background context.
58:45
Of course, then I love it. It's so perfect.
58:47
So the thing that I was thinking about is that this feels like it
58:52
is a. A new type of benchmark, right?
58:58
It's not like the the benchmarks that you see or the evaluation
59:03
benchmarks that you see necessarily.
59:05
Like, hey, how's the output? It feels like there's this whole
59:09
new content or it's, I don't know what you want to call it.
59:14
Yeah, context window benchmarking and how well the
59:18
context window holds up, right? And.
59:23
There might be something there that that could actually be a
59:26
useful tool for people to have as they're testing out their
59:30
different use cases and if the. Whatever the tool is like.
59:38
If you know that. You're gonna get a higher hit
59:42
rate. If the.
59:44
Fact is, in a certain part of the context window, then you're
59:49
going to want to manipulate your text or your prompts or whatever
59:53
it may be so that it coincides with that, right?
59:57
Yeah, potentially. You know, this is a little bit
59:59
of a hot take here, but if you're passing 120 thousand
1:00:02
tokens where the context of GT4, you're being a little lazy.
1:00:06
And what I mean by that is you gotta go do better retrieval.
1:00:09
Who in the right mind needs to pass that much because they're
1:00:12
they have that much uncertainty as to where the right thing is.
1:00:15
Now I know you can do it for summarization, but like, come
1:00:18
on, that's a lot of kind. You know we don't need to
1:00:22
summarise Moby Dick. No, that's 130 pages.
1:00:25
Like there's a hundred freaking. I think they might even said 300
1:00:28
pages where this stuff just. In that Oh my God, if you wanna,
1:00:33
if you wanna rag on me for San Francisco.
1:00:34
I mean, even the fact that we're laughing at that is just, that's
1:00:38
tech boy stuff. Yeah, exactly.
1:00:41
By the way, that's so true. I went and I looked up
1:00:43
modelbenchmarks.com and actually was free.
1:00:45
So I just got it. Actually so.
1:00:47
No, we haven't. No, you bought it ohh which you
1:00:50
have it too. That's awesome.
1:00:51
We can collaborate on. That is incredible.
1:00:55
I love it. Yeah, and the best I'm.
1:00:59
I really feel like that was something I was hoping hugging
1:01:03
face would do better, but it feels like hugging face didn't
1:01:07
really the leaderboard and just what you get on hugging face
1:01:12
isn't super clear. And maybe it's because there's
1:01:15
just so much material out there and there's so many.
1:01:20
New models that are coming out, but it would be incredible if
1:01:24
it's like Ohh hey. XYZ super niche thing.
1:01:28
Here's the model for that. And there was a consensus.
1:01:32
I guess. It's really hard to do though,
1:01:34
to have everyone agree that this is the model for that.
1:01:39
Also, there's a few things. You have to agree on what the
1:01:42
version of the model is that you're running.
1:01:45
And then two, you have to agree on the benchmark that's being
1:01:47
used. There's a bunch of benchmarks
1:01:48
out there and everybody wants to claim that they have the right
1:01:50
benchmark. So, but I think it's a I think
1:01:55
it's a problem worth investigating.
1:01:57
And then sure, you have a nice and beautiful UI that's on the
1:02:00
website. But then if you apply that to,
1:02:03
if you give that via API, maybe there's some dynamic reasoning
1:02:07
that you're doing based off of what's trending.
1:02:09
Well right now I have. I have no idea.
1:02:12
I shouldn't. I should have built it out first
1:02:13
without the domain. But who knows?
1:02:15
Where Shooting from the hip here, man.
1:02:17
This is a new video idea. Exactly.
1:02:19
Love this idea. Because yeah, you can make there
1:02:23
is. Have you heard the idea around
1:02:26
the API Gateway? And like having some kind of
1:02:29
proxy that will route your prompts or whatever it may be
1:02:34
like your API calls to the to the right model or to the right.
1:02:39
It might not even be a model that's needed, right?
1:02:42
Depending on the call. And so you have something that
1:02:46
is is a gateway and it can say, oh hey, well, I see this prompt.
1:02:51
And because I noticed that this prompt is dealing with some kind
1:02:56
of legal jargon, we're going to route it to the fine.
1:02:58
Good legal LLM or I see this prompt.
1:03:03
I don't really understand what it is.
1:03:04
Let's just throw it at GPT for and have this, Yeah, I mean it
1:03:08
makes it. That's a like a routing question
1:03:12
and I don't see we haven't yet gotten to the world where I my
1:03:19
hypothesis is we'll go this way where there's a small fine-tuned
1:03:24
and models for niche use cases. I think the world going that
1:03:28
direction. We haven't yet, I haven't seen
1:03:29
that in wide practise quite yet. Everybody has their own internal
1:03:32
models. In fact, I think that's gonna be
1:03:35
a big thing that we didn't hear from opening IDEV Day that I
1:03:38
think we'll hear next year is, or at least by next year is you
1:03:41
don't have the ability yet to power a GPT by a fine-tuned
1:03:46
model. I think that is going to be
1:03:48
really, really cool because you'll be able to get extra
1:03:51
performance out of your GBT because right now you can just
1:03:55
pick like you know, GT 3.5 or 4 or whatever.
1:03:59
So I'm looking forward to that. Yeah, I thought I I didn't
1:04:03
realise that they didn't announce that.
1:04:05
I thought that was just kind of like inherent because it feels
1:04:08
like you're going to be able to get so much more out of it if
1:04:12
you can have a fine tune and your own special, like you add
1:04:16
your special sauce to this fine-tuned one and then you
1:04:18
throw that on the marketplace. Yeah, Yeah, yeah.
1:04:21
I mean, I agree. I think that we will see that
1:04:26
for sure. And I think that the marketplace
1:04:28
for those fine-tuned models will be extra good.
1:04:30
So yeah, anyway. Yeah.
1:04:36
Well, I'm looking forward to it, man.
1:04:38
I appreciate you coming on here. This is awesome.
1:04:40
I love talking to you, as always, and I encourage everyone
1:04:45
to go and check out the video to see where the best place to put
1:04:51
the truth is in your contacts window.
1:04:54
Yeah. Ohh, it's so good.
1:04:58
I appreciate it, man. Yeah, absolutely.
1:05:01
Well, thank you for having me on.
1:05:02
And I'm a big fan of the ML OPS community, so thanks for
1:05:05
everything that you do there. This is Skyler.
1:05:12
I lead machine learning at Health Rhythms.
1:05:14
If you want to stay on top of everything happening in ML OPS,
1:05:18
subscribe to this podcast now. Now now.
1:05:25
Now now. Now.
1:05:30
Now.