Latest / Agentic Conversations (formally mlops.community) / Large Language Models at Cohere // Nils Reimers // #158

Transcript

Large Language Models at Cohere // Nils Reimers // #158 · Agentic Conversations (formally mlops.community)

0:00
Hi. I'm Neil Stuart of machine
0:02
running for crew here. I'm not a coffee drinker, which
0:04
surprised a lot of people but more like hot chocolate.
0:07
So this keeps me awake. Especially in Winter.
0:10
Get it got hot chocolate and the morning and then your app and
0:14
ready to do more machine. I welcome back to another
0:19
envelopes Community podcast. I am your host Demetrios and
0:22
today I'm joined by none other than Abby.
0:25
What's going on? A be fantastic.
0:29
Very interesting. Conversation today that Liars
0:33
used. I forgot how he pronounces - I
0:37
actually, like the idea of Niles.
0:40
I was calling him Neil's, but I think at the end of the day, you
0:44
could go for both. I mean, who is Nils or Niles?
0:47
Is the director for machine learning.
0:49
Echo here, very interesting background.
0:52
So he started working in machine learning research, but very
0:56
specifically and the pp search back in 2014.
1:00
The Field Force in that hot, there is some work around guns
1:05
which were starting to come up in the field or since then he
1:09
has done some pretty amazing work around your search spaces
1:13
around, but it will search. I think one of us worked, which
1:18
is sentence word became quite popular in the community, and in
1:22
the past he's worked with hugging face and no, he's with
1:25
go here. So very, very interesting
1:27
conversation. Very interesting person to have
1:29
this content. So I really enjoyed his whole
1:34
outlook. I mean, let's let's preface
1:36
this. He gave us a ton of information
1:38
about what coach here is doing and cohere.
1:40
I was very interested in talking to because they're one of the
1:44
few companies that I feel like are actually doing lme's in
1:50
production at a large scale. And it's the different kind of
1:55
production than I think a lot of people are actually doing lme's
1:59
in In and I use my little fingers to do the air quote with
2:03
that because there is different types, right?
2:06
And this is something that we talked about a week ago, on the
2:10
llm, in production Roundtable and one type is just building on
2:15
an API. Another type is maybe
2:18
potentially chaining together, a few API calls and maybe hot
2:22
swapping models or or apis that you're using.
2:27
And then the other is getting these models.
2:30
Owls and really going in and fine-tuning your room and
2:33
actually, like doing the work or building the model yourself,
2:37
which is what Co here is doing and training that model from
2:40
scratch. And then they're the ones that
2:42
are exposing these apis through the rest of us.
2:45
And so I think that's fascinating to hear like what
2:49
the challenges are and I love that he went over.
2:51
We ask them some of these core challenges that a lot of people
2:54
have been talking about one being cost and for them when
2:59
they think about Cost, they're not only thinking about the cost
3:02
of serving, these models to the public through their API, but
3:05
there's the cost of actually training and Staffing up.
3:09
The people that know how to create these models and and then
3:13
the all those GPU resources and all that good stuff.
3:17
And then the other ideas latency, that's another huge
3:21
one. So he went over both of those.
3:23
I thought those were excellent. Answers.
3:25
Was there any other key takeaways that you enjoyed from
3:28
this conversation? I loved To DARE productive
3:31
Focus, very specifically, when we asked him the question which
3:34
is are you focused on how do we build a bigger foundational part
3:39
of like this point time? And what are you reading in that
3:42
specific area, or are you focused on the apps?
3:45
And everybody was building products on top of the models
3:48
that you have. So that was one conversation
3:51
which are left with, and which she said, you know, it's time to
3:54
be because we have to make money out of these models.
3:57
It makes perfect sense to be able to The word applications
4:02
are people using them for and pray flies to that also the
4:07
conversation around security because that is one thing that
4:12
enough people are talking about. So those were two heights,
4:16
Family, Security always gets the short end of the stick know,
4:20
there's never enough people talking about it until there's a
4:23
huge security breach or hack or something and then it comes up
4:27
for a few days and then it goes away again.
4:29
But there is this predicament that I think a lot of these
4:33
foundational model providers are in one being that they're
4:38
competing against the likes of Googles.
4:40
And Facebooks you have basically endless resources to put out
4:44
these foundational models that they're creating and they can if
4:49
they choose. So like we've seen with Facebook
4:52
recently, right? And Google in the past they can
4:55
just open source those models and then they're also competing.
5:00
The likes of open AI which is has really just taken off.
5:05
And I think if most people are building on top of large
5:09
language models, these days, it's probably something to do
5:12
with GPT 3gpp, for whatever the new GP T is.
5:18
By the time this gets released potentially there's or igbt 20,
5:22
but with our release scheduled, but the idea there is that I
5:28
think it's it's fascinating because Cuz they don't like the
5:31
Facebook's and the Google's have the foundational models as
5:37
almost like a side business. Just to stay up-to-date and stay
5:40
current and all the R&D that they're doing is something fun
5:45
but it is not core to the business that is just printing
5:48
cash. Yeah and so coheres if I could
5:51
lie it's it's not the prettiest exactly.
5:54
And yeah, and when you look at some companies like cohere or
5:57
and propagate or adapt, these companies needs to figure out
6:02
the business model around the large language models, which is
6:08
a whole different game and so I'm fascinated to see how that
6:11
plays out in the future. And when you think about this in
6:16
two years or you think about this in five years, what is that
6:19
going to look like in? So I think noodles brought up a
6:21
great Point around that when we I asked him, what's your
6:24
differentiating Factor? How are you at?
6:27
Cohere, trying to separate yourself from the rest of the
6:32
foundational model providers. And I felt like he had a spot on
6:38
answer for that. I don't want to spoil it for
6:41
anyone, so I'm just going to leave it at that.
6:43
Listen, it's kind of at the end when he talks about it in joy,
6:46
and if you are not already signed up for our newsletter, if
6:51
you're not already going to one of our local, you'll meet ups
6:54
that are happening in all of 37 cities around the world.
6:59
So there's probably one not too far from you and that's it.
7:02
Let's get into this conversation with nails and I took care of
7:04
us. This refuse otherwise to beat
7:06
useful personalized, message you and chase you to write a five
7:09
star review for us. That's so true.
7:12
The people underestimate my ability to be Petty, so I will
7:18
find you. We love reviews and we love it
7:21
when you write funny reviews, if you have that possibility, it
7:25
would mean the world to us. Let's Nails, here you are very
7:36
close to me. I'm surprised we're not doing
7:38
this in person, actually, because you're only like, 40
7:42
minutes down the road from my house where I can look at this.
7:47
I'm in this little town right. Nearby.
7:50
Schwalbach and you're in. You're in Frankfurt if I'm not
7:54
mistaken shopping even, so it could be unclothed.
7:57
Yeah. Even closer, dude.
7:59
So yeah, we'll have to meet in like Wiesbaden for some pizza or
8:05
something sometime and and I'll practice my horrible do each
8:10
with you. But today we're here to talk all
8:15
about large language models. I think it's worth, just going
8:18
over a bit of your background because you've got some
8:20
interesting pieces. I mean, you did a bit of it
8:24
security that I want to get into and how that relates to what
8:26
you're doing these days. You went and did a PhD but all
8:31
stop talking and let you tell us what exactly your history is.
8:35
He had already appeared to be here.
8:38
Yeah, as I mentioned, I did a lot in the past, so so I kind of
8:41
like like to Pivot and do like every five to ten years.
8:45
Something different. Originally, I started in web
8:48
development during my lot of software as a service business,
8:52
and creating my own companies. And then co-founding companies
8:55
was people in the early. 2000's when software as a service was
9:01
not like a business concept to a lot like an unknown term after
9:05
that, pivoted into it security, a lot of cryptography, a lot of
9:10
ever Break Stuff understanding security systems, how to set it
9:15
up, how to test the total pentester.
9:17
No holes are to break it and then in early 2013, pivoted to
9:24
Ai and machine learning which was completely different field
9:27
at that. So so what?
9:28
Like they was like, basically no business interest in machine
9:32
learning only relief you. Maybe Google is doing like,
9:35
machine learning at that time, I was like just get an interesting
9:39
field man. Really in the past ten years,
9:41
totally changed. Now you see I want talking about
9:45
my grandparents know about Chad CPT and just transformers stuff
9:49
like that because they write it. I don't know in the Time
9:52
Magazine was so long, like some news, show him on TV.
9:58
And you are really happy to be in this field.
10:00
Yeah, there's a bit more hype around it now than there was in
10:03
2015. I had definitely says out
10:07
especially annulled P like when I started no one.
10:10
No. Okay.
10:10
What's NLP and why do you use it?
10:13
Some said, yeah, maybe named entity recognition on eels
10:16
articles that's interesting to know which companies are people
10:20
talking about, and how is perception of I down.
10:24
JPMorgan versus Bank of America, and no one was there.
10:28
I'm really interested and I'll pee and here up like last year
10:31
this year. Now I want to ask about line.
10:33
Every language processing And it became the hot topic for sure.
10:39
So what prompted you to make that switch from it security to
10:44
machine learning. Yeah, so I finished my Master's
10:47
Degree and it security where did like a lot of theoretical stuff
10:52
and cryptography doing like certain proofs about security
10:56
conditions that cannot meet at the same time and a challenge
11:00
was security is it's really abstract so fast.
11:05
You you spam Manny I don't know how much dollars for McAfee
11:09
antivirus software, someone who Iris Software was this hundred
11:13
dollars like a worthwhile investment for you.
11:15
And in the past year you say I don't know, I spent $300.
11:19
Was it worth it soon as bad? Next year?
11:21
Two hundred dollars on it. Should I upgrade?
11:24
So I really, really had to say because security there's like
11:27
none. Really?
11:28
Yeah. You you cannot measure it like
11:29
is there like a return for you? And why I said, okay, I want you
11:33
to go back more Hands-On and say go in a few.
11:36
Well, you see that's valuable. So it's like kind of to build
11:40
something, which you can touch her and says so went back to my
11:46
machine learning had in the University already cried class
11:51
in Berkeley introduction to AI, where Professor Dan Klein was
11:57
like introducing all the concepts on Pac-Man.
12:01
So like the whole semester were like programming Pac-Man's,
12:04
making Pac-Man smart back. King the ghosts murder.
12:07
So how can the packman automatically oval, like, avoid
12:11
The Ghost and eat all the, all the bubbles and also do like
12:16
tournaments where you have our two teams competing, each other
12:18
as and who can like fight the other teams and eat all the
12:21
bubbles from the other side of the board which was super fun.
12:25
And kind of had like these feeling you build something
12:28
which can touch. So you see, you see how the
12:31
agent starts, really stupid and runs into the girls and I will
12:34
see every iteration you make it. So Martin spar.
12:38
Why said it's yeah, super interesting, as an engineer, on
12:41
the one side as a software developer for me, one side and
12:45
that time. So, in early 2013, I said, yeah,
12:48
if a model, if computer, can really understand text, it will
12:52
be extremely powerful. So so we have all of our human
12:55
knowledge captured in text. So if you think about like, how
12:58
much nourish tears in Wikipedia, on all written books on all news
13:03
articles. If the computer understands this
13:06
and really has read every Every single Wikipedia article and be
13:10
able to understand it like, how smart can the computer V?
13:13
And I think it's always good if you're smart, or if your
13:15
commuter smart, that's always valuable in our society.
13:19
So I was really interested to dive into that so true.
13:23
Yeah, it's parked that flame inside you and it's funny.
13:28
You say that the it and Security in general, is it as tangible?
13:32
Because I could see that like, oh, yeah, did it work?
13:36
Did McCaffrey Is actually do anything.
13:39
Or is it that I just like, was not targeted and I guess you
13:43
don't realize you need security or the beauty of having these
13:48
Security Programs, until the day that you get hacked and then you
13:54
can put a price on it. And then you're like, oh man,
13:56
I'll pay whatever. But yeah, you never know.
13:59
If you have to catch a wireless get hacked, like, would it have
14:03
been changed if you watch have upgraded to the more expensive
14:06
until virus off? We're and also, and if in a
14:10
business context, it's even more challenging if you if you're a
14:13
bank discussing should we upgrade or encryption standard
14:17
go from 128 to like 256 bits keys?
14:22
Will it make a difference? You don't know.
14:24
And, and that's what's bit sad for me.
14:26
So, I really like, I want, it's tangible and you can say, hey,
14:29
that's that's amazing. That's people want long.
14:31
They do use it, they love it. The, the usage increases so
14:35
that's what I like to do. That more to do, but you
14:38
obviously security super relevant and also, in terms of
14:41
like language models also really good, hot topic, interesting
14:45
field. That's exactly where I was going
14:47
with this. Like, how are you bringing what
14:50
you basically? Have innate.
14:52
I imagine from all those years being in the security field.
14:56
Now in what you're playing with with machine learning and more
15:00
specifically NOP, you did, you see a lot of challenges coming
15:03
out was these flash language models.
15:05
Generate from models on the one side.
15:09
If you train them, they're like, certain challenges like data set
15:12
poisoning. So, so people try to sneak in
15:15
there, like, harmful content into the training data sets.
15:21
No information is a challenge. If you have like these
15:24
generative models like, hey, what's the private number phone?
15:26
Number of that person should not reveal the like this number and
15:30
external factors for people, misusing the models to create
15:33
like personalized span to E-flat Google to put advertisement on
15:38
it or to do like targeted phishing campaign.
15:41
Say okay. Yeah, I follow your comments on
15:44
Reddit and now I send you like a targeted automated email
15:48
phishing email on this. This has many Interesting,
15:52
upcoming security challenges Medfield.
15:55
Mmm. There's like not only like the
15:58
security that's happening inside of the models but also how I
16:03
imagined Bad actors can just augment what they're doing with
16:07
the Newfound capabilities of these large language model?
16:11
Yeah, I mean that the goal of T / language model just to make us
16:15
more productive to automate your boring uninteresting task away.
16:22
Like, I don't know writing reports or I don't know, maybe
16:24
you have like different bullet points but you have to make and
16:27
then I like nice long written application because that's kind
16:31
of like socially expected. But you're not sending like
16:34
three bullet points where you're amazing but I have all these
16:36
boilerplate when you apply for a job.
16:39
But yes sadly it also makes the Bad actors and people with bad
16:42
intent more productive. And and so that's always like a
16:46
balance. We have to find this as a
16:49
society. How can we Make the goods,
16:53
people more productive and like try to reduce the risk of
16:57
making. Also, the bad people more
16:58
productive and is that something that you're focusing on right
17:02
now, as you've been working in, large language model, especially
17:06
like, because the school has increased massively and there
17:11
will be building applications on top of them.
17:14
Or was this also a concern for you, while you were doing it,
17:18
like your PhD then I'm doing the PHD at was not so much of a
17:24
concern. So so during my PhD at was more
17:27
like text understanding like find, I don't know, maint
17:30
entities and news articles and they're not so much bad intent.
17:35
Can be derived but we agreed it was these generate models.
17:38
You see a lot more interest and it could hear.
17:41
It's also like a big big interest on the one side.
17:44
How to avoid that? Harmful content gets into the
17:48
training of foreign models that personal Educational Credit
17:52
Card, Security numbers, get into the mall and we produced by the
17:55
model. How can we avoid that?
17:57
The model is not being used to like hate speech, producing hate
18:03
speech so that you flooding social media was like, generated
18:06
hate speech that but it's kind of like hard and trade-off
18:11
because also security principles.
18:13
There's a trade-off, for example, people won't like data
18:16
privacy, so they don't want that model providers spy on what How
18:21
are they using the models? But what use cases, and what
18:24
text do, they produce with it? Because cannabis F like
18:28
sensitive information in it, but if you ensure like these
18:32
privacy, you cannot find like Bad actors.
18:35
That's creating like a phishing attack and try to mail million
18:39
users. 10 million users was like really personalized span to do
18:43
like fishing and try to get like credit cards from it.
18:45
So so you can you have to do the trade-off, like ether, you will
18:48
respect the privacy. Now after users, then you can't
18:52
avoid like the batch actors or you will say, okay.
18:55
You wild more like on privacy and then you can prevent better
18:59
at like piece, what identify these Bad actors that try to
19:01
misuse the models, from a society perspective and
19:06
regulation perspective, it will be interesting how to deal with
19:10
it. Like, how to go forward also
19:13
from a such site perspective. How do we need to evolve like,
19:18
with every technology, they Misuse like my email came up
19:22
spam. Oh yeah, I'm up.
19:25
So Twitchy which the didn't know from like classical written
19:29
matters but yet until evolved? No.
19:32
Okay. Yeah.
19:33
I should not trust like every email I get.
19:35
So if I get like an email presumably from my bank claiming
19:39
hey please enter your password Here We hopefully learned as
19:44
humans to not trust it and then we just need to toss evolve and
19:48
see okay what camera Trust and what can we trust and well-liked
19:52
near drastic measures? It can Implement here.
19:56
Dude, so there's something else I wanted to ask you about
19:58
because you basically were hanging out in Academia for a
20:01
while, and then you moved on to Industry and what made you want
20:07
to take the plunge into the industry and start getting your
20:11
hands wet, like trying to make money with machine learning as
20:14
opposed to try and write papers about machine learning.
20:18
Yes, I love building. Stuff says so that's that's what
20:22
I did for 20 years. Well, we on be like a really big
20:26
carnival direct even like 20 years ago for open source
20:28
community. Maintain and manage him set up
20:32
so many open source projects and Yan in science at some point.
20:37
When I make the move is head of. Yeah, it's I like more on
20:42
building stuff than writing the papers.
20:44
So writing really scientific papers is a lot of overhead and
20:48
it's not really focusing on what's useful and what's helpful
20:52
for users. So so you or the incentives is
20:55
different. So you have to write the paper
20:58
and really specific format so that it gets Accepted, but it's
21:01
not really. What's helping the use of the
21:04
most and often what's happening use of the most.
21:08
It's not publishable. So for example, one example in
21:12
the field is like the reporter paper from Facebook where people
21:17
team from Facebook was working to make bird better on the
21:20
Stanford better. A trend it on more tokens, get
21:23
out like a better model which is used by millions of companies
21:27
the Roberto model so extremely helpful but when the Submitted
21:30
it has a paper the scientific Community set no way to rejected
21:35
from this conference because it's not novel, it's just
21:38
burning rain on more tokens and optimized and I didn't like this
21:44
disparity that you say, okay this is like what really helps
21:47
and has a massive impact on users and it's already ordered
21:51
by the scientific community. So and I said, ok, let's go.
21:54
We build stuff that's helpful for users and don't care if
21:57
three random And analyze reveals like it or not, and decide if
22:02
the paper will be accepted or not. here are interesting and
22:09
Something you need to talk a little bit about before this and
22:14
you said, you know, one of the main things that we need to look
22:17
for, is how our users interacting with these models.
22:23
And what kind of information are we Gathering using these models?
22:26
Because we're constantly training the model on the
22:30
information, or on the type of use tsys, that people are using
22:34
these models bird. So, one of the things that
22:37
happens in, this is something you also get.
22:39
Talk on which is basically Still Waters are a major issue.
22:44
So do you think that and because yes we're working with deep
22:49
learning and one of the major flaws and deep learning has been
22:53
catastrophic. Forgetting, do you think
22:56
catastrophic forgetting then becomes a feature or a flop 2012
23:00
when you moved? Yes, sir.
23:02
So that's interesting, topics. Many aspects on this first.
23:05
Like, stay of models is a challenge.
23:07
If you ask the bird model, And Bert doesn't know anything about
23:12
Corona. So still thinks coronavirus a
23:15
beer now. And but if you're awesome model
23:18
like how do I catch Corona? Well, what's the transmission of
23:22
coronavirus those things? Yeah.
23:23
How do I catch a beer and sadly? Well happily doesn't know like
23:27
about connection to covid-19. Also, if you put in like in a
23:31
generative models and I say, hey, write like a Write an
23:36
article about the political career of the u.s. president and
23:40
then the Board model still thinks that Barack Obama is the
23:43
US president. So it outputs you like to eat
23:46
politically click here for Barack Obama up to like 2017 or
23:50
2018, which is kind of frustrating.
23:55
So, some big question in the fields, like how to update it
23:58
and catastrophic forgetting is in general, I challenge all.
24:01
So, when we add more and more capabilities to it, but as we
24:05
see as the models become really large, they typically have like
24:08
enough capacity. So, so if the models have like,
24:11
ten hundred billion parameters catastrophic forgetting is not
24:16
that much of an issue, where you say, Okay, Adelaide a new
24:19
feature. Now it can summarize what can
24:21
code, or I can talk like a pirate.
24:24
It's not really forgetting that much of the past, but really the
24:28
updating of the knowledge of this is tough because we're
24:32
humans were super efficient. So this is when you reach when
24:36
New York Times, I don't know the next u.s. president election.
24:39
This person is like the new newly elected president.
24:42
We can update our knowledge from the single example, but these
24:46
generative models aii models. They sadly have to see like
24:50
Millions 10 of these occurrences take A lot of data topped at
24:55
once. Single simple fact about like
24:58
who's the current u.s. president.
25:01
We're also really good in updating Making Connections.
25:03
So we know this person's Anil US president.
25:06
So we know K. First lady is changing was like
25:09
the ruling party in Washington is changing.
25:14
So we really get to make all these connections from like I
25:17
don't know five for its at the generative model, maybe can
25:20
update the new US president but then still brings the First lady
25:25
this deal, Warren War doesn't know, like, who's like in charge
25:28
of the White House. So what I'm more interested in
25:31
is like really combination of like search Technologies and,
25:35
and generative models. So that's an exciting technology
25:39
where we currently and these big models generate for Mortals.
25:42
We have kind of stored all the world facts, so they know about
25:46
like every country, every Capital presidents and makers
25:50
and I don't know, Hector's in movies.
25:54
And so on. Which what's my going to really
25:56
hard to keep it up to date and I think an interesting trend is
26:01
life where you say, okay, will you train these generative
26:03
models just to be good on reading comprehension so so can
26:09
we text and then listen on the text and then plug in like a
26:15
good search components. So if I have like, hey, write an
26:18
article about the u.s. president, it would first go on
26:22
Google search. Find currently As president open
26:26
Wikipedia, see? Okay?
26:28
This is the Vicki. Periodical of the current u.s.
26:30
president. And then, based on this article,
26:33
produced the output you mentioned, the two threes of
26:36
going about the solution. One is we train the next model
26:40
on lots of data. And one of the main reasons for
26:43
that is because these are probabilistic models.
26:46
So the probability depends on the number of Articles.
26:49
It has already seen in the database that to resonate with
26:53
that kind of result. And the second is basically
26:56
something you said which is the search as a way to be able to
27:00
update your information. Which categories is it better to
27:07
use one approach over the other when we are incorporating this
27:10
new knowledge into large language models, bowls are still
27:14
like really golly. So it's not like really that the
27:18
one or the other works. Well, me personally, I like the
27:22
search one more because it's really It updates a lot quarter.
27:26
So so when you think about how quick editors on Wikipedia are
27:30
and be able to change anything so I don't know.
27:34
Like I don't know and Monarch like Queen Elizabeth passed away
27:37
last summer with in a second. It's updated on Wikipedia and
27:43
it's not really feasible. If you try to put this into the
27:46
model and update it because it's only like a single data point
27:51
that Queen Elizabeth passed away, so it's not ready.
27:54
Just as okay that that's noise I will ignore it until I see it
27:58
like a million times and it takes really long until you got
28:01
like a million articles talking about, Queen Elizabeth passed
28:04
away, it was something more minor.
28:08
So I don't know, there is like an election and some tiny City
28:11
and to make sure it's changing, it will take a really long time
28:14
until you have enough text and wizard.
28:17
She kind of always if you have searching next up to date.
28:19
If you always pin to like that most up-to-date Wikipedia You
28:25
have like these instant updates. So as soon as you like one
28:30
second after Queen, Elizabeth has passed away, you asked the
28:33
model, hey, can you write me a summary about the life of Queen
28:37
Elizabeth's? It's band can produce the
28:39
output. She passed away on whatever date
28:42
it was. One of the things I think of
28:44
when thinking about this is basically, how do we Define two
28:52
You wrote search space and build adaptive Waters that are able to
28:57
adapt a different kind of scenario.
28:58
So one of the reasons for that is it depends on let's say if
29:02
we're making a search, which is information retrieval and what
29:06
kind of information retrieval is whether we're looking at facts?
29:10
Or are we looking for? Let's say code, and for code
29:14
bases, again, for the kind of use cases that gbt is trained on
29:18
is perfectly reasonable to be looking to be able to Look into
29:21
like the historical data, instead of like the new
29:24
Fertitta, I don't think it makes that much of difference or to
29:27
search for like, what's new on Stockton field.
29:30
Yeah. What disagree on this tunic,
29:33
including historical data? Yes, and certainly use cases.
29:37
It's fine. Like, if you think about like
29:41
perimeters in Python like, how do I Define like a function, if
29:44
statement a loop that's get okay to have these historical data,
29:50
but then if you work Was apis and Frameworks may like new
29:54
versions coming out and then maybe you sound popular
29:58
framework. I don't like Sky, could learn if
30:01
it's putting out like a new. Major version will pie torch by
30:03
all like were Shinto. And you ask like hey how can I
30:07
create like this module in Python watch maybe in the
30:10
historical data just now is like from Russian one, how to do it
30:14
with a test doesn't have like these up-to-date.
30:17
And then it would be nice. I mean, that's what we as humans
30:21
also to. So, we kind of, remember,
30:23
something like how to do basic stuff, but also a lot of things
30:27
like Googling checking out or taking the IDE and see.
30:31
Okay, what were like the parameters again for this
30:33
function and Pie torture and my favorite framework and I know
30:37
also yeah, it's nice because we see okay.
30:39
This is the version I'm using or that's the latest version.
30:42
We can direct the updated as a unit so I'm not.
30:45
I would not advise. My Engineers to say, hey
30:47
remember I memorize and we think on these tutorial on the
30:51
documentation of pie torch, I say no learn how to search what
30:56
to search for. When do you need to search and
30:59
inject Research into account? And that's what we do was
31:02
generate of models. So at crew here, we're teaching
31:05
them how to search like like really be able that they know.
31:10
Okay, this isn't Gap. I should check use Google, use
31:13
em Anderson search, use internal search to And the relevant
31:17
information pulls the right search query and then be able to
31:21
reason on what you find care. It feels like that is a really
31:24
good use case for putting some kind of catching model before it
31:30
actually goes to the large language model, where this model
31:34
can filter out should this be searched for or should this go
31:39
directly to the large language model?
31:41
Yeah, that's that's a challenge in terms of pre-training says so
31:45
right now Free trainings like just next token prediction so so
31:50
I say the current u.s. president is and then the more Noah's Ark
31:54
I need to predict the next token, which is already a
31:58
challenge in the training data because I don't know.
32:00
From when is the training data them, is it not It Go from I
32:05
don't know 2005 for 2015 with 2023 and then also I like with
32:11
Sturridge bring in the pre-training to now.
32:13
Okay, what should we search for? So it's still Many, many open
32:18
questions. How can you also adapt The Surge
32:21
to the respective time? So if I pre-trained on article
32:25
from 2015, how can I have like a snapshot of Google?
32:30
For example, from 2050, just like, think of threw this at my
32:33
head. It would be very hard to know
32:36
what these topical items are considering that they're always
32:40
changing. Like you said, like, Corona all
32:42
of a sudden, something like Corona comes up and where as
32:45
before That wouldn't have been something that the model
32:50
probably would catch at. Now, it needs to be.
32:54
And so you need to figure out those ways to quickly switch in
32:58
these topical issues, and make sure that you can stay
33:02
up-to-date changing gears a little bit.
33:04
I know that one of the other very hot topics when it comes to
33:09
dealing with large language models, and actually, like using
33:13
them in production, is the latency issue.
33:17
And we had sahil on here from you.com and he was talking about
33:22
something that I thought was fascinating, which is the actual
33:25
non-subjective time that it takes to get some answer back
33:32
and then the perceived time and they're doing all kinds of cool
33:37
tricks to make the perceive time, feel shorter.
33:40
So maybe that's they give you some kind of quiz while you're
33:43
waiting for the answer to come back or they tell you.
33:47
Some fact or etc. Etc.
33:49
So it occupies you and then you go.
33:51
Oh, look at that before, you know it, there's my answer.
33:55
I'm wondering about what are the technical things that you all
33:59
are doing to make sure that the latency stays down?
34:03
Because if you're not very, very on top of that, it can get out
34:08
of hand really quick. Yeah.
34:10
So so that's a shadow aspect models and has a lot of
34:14
Hardware, requirements and relax.
34:17
Via your what you say a perceive time, actual time.
34:20
So one feature we recently launched is like streaming so
34:24
say you say right near blog post about the monarch of England and
34:29
then so, so before that we were like producing all the output
34:32
and then send the output as a chunk back and say, Here's your
34:36
article, we change it to a screaming.
34:38
So so as soon as the word is generated, it's like screen back
34:42
to the user. So you can stop leading a menu
34:44
CLK, the article starts to ride. Within a split second.
34:48
You can already start reading and then the latency perceived
34:51
latency is a large smaller ISO. So if the model can produce
34:54
Quaker worse than you're able to read, you're not receiving any
34:58
latency. So so you're not fighting
35:00
especially with long generated text like hey you have to wait
35:03
like 10 seconds, 20 seconds to get the article but you see.
35:08
Okay it's producing you can start to read and you probably
35:11
need more than 20 seconds to read it anyway but that's only
35:14
one side like the trade, psychological tricks.
35:16
You can make it to proceed faster, other sides to to make
35:20
the model to do. We actually faster first is like
35:26
model size so so the smaller the model the better but this was
35:32
like in Conflict was the quality of the model where we see, okay,
35:34
light your models are totally more capable, Hardware
35:39
optimizations. So the field moved away from
35:43
float32 to float16 is normal. Going to like in Dade.
35:48
Also in video was the hopper h100 gpus showing float 8.
35:56
So where you see, okay, Flo Dade is think something like four
36:00
times, or even more faster than float. 32 And then there are
36:05
certain other tricks for machine learning Hawaii, run like two
36:09
models in parallel like a small model that's great.
36:12
Like, first draft of the article and a second model, the latch on
36:16
oil, dad's kind of rates rates this and takes the input from
36:22
the smaller model as the input to navigate the search space.
36:26
And and there we also see like get improvements from that but
36:30
in general, I challenges like yet deploy it.
36:32
And how How well does it work with like perceived a time?
36:37
Especially in a streaming setting, where you say, okay,
36:39
maybe this optimizes the end-to-end time but it's not
36:45
comfortable with like streaming so I whistles perceive time.
36:49
So there you have to find that trade-offs again and other quick
36:53
question because you said, you would have two models running in
36:57
parallel for daily running in parallel, creating two different
37:01
outputs at the same time or are you thinking About using them as
37:05
like an actor critic. Says so challenge, was he
37:08
shouted for models, is you have a token and then you predict at
37:12
the alport, part of these 250,000 tokens or whatever your
37:18
vocabulary sizes. Which one is the most probable
37:21
next morning. So that's kind of a challenge
37:24
that you say, okay, if you have like these 250k tokens at the
37:28
output to really say, okay, these tokens should be the next
37:31
one. And here using like a Smaller
37:34
model to not predict like what all I did 250k tokens but you
37:40
said okay use a small model do the projection and then you go
37:43
like the top 10 tokens from a small mole and a larger model
37:47
just needs to see okay from these top 10 tokens which is
37:50
like the best one. Okay that makes that makes more
37:53
sets and for problems while that's understandable for text
37:59
vertically, the big problems according to win, peaced out.
38:02
Going into the The territory of, let's say, multilingual, search
38:06
because search, and just like a single language is a little bit
38:10
easier, but when we have more Vitality is our, when we have
38:13
World languages, it becomes a little bit more complex link.
38:16
This search space becomes exponential, for generative
38:20
models to believe what you do is like increase the vocabulary.
38:25
So so often famous. You have something like I don't
38:29
know. 50,000 tokens you're searching over.
38:32
And then in the morning, we'll setting you increase it to like,
38:36
and another 500,000 tokens. But it's still a still do above,
38:43
so let's keep moving and I want to dive into a lot of the work
38:48
that you're doing right now at Co here.
38:50
And let's just start by laying the foundation of what it is
38:53
exactly that you're doing it, go here and maybe even for those
38:57
who do not know, because I know there may be some people out
39:00
there that have not even heard of cauchy or so, can you give us
39:03
a breakdown of what Co here is and what you're doing there?
39:06
Great. Yeah, sure.
39:07
Soco here is Canadian company founded back in 2019 from one of
39:13
the co-authors of the Transformer papers, agent,
39:17
Thomas and and France, where they saw like a lot of
39:22
transformation was in Google, though, the Transformer
39:24
architecture in the bird architecture, hat and said,
39:27
okay, yeah, there's just should be more widely accessible to
39:31
users. And cohere, we're training.
39:33
Foundational models in NLP. One is generative models.
39:38
You Give text and light. Write me a blog post about
39:41
analogs and focus on security. And the others text
39:47
understanding which I'm in charge of be a little till on
39:50
the one side surge on text recommend text, while I
39:53
understand what's written in the text.
39:55
Let's say, you have customer reviews, 10 million.
39:58
Customer reviews, say what do they like about your company?
40:02
What do they dislike about the company?
40:03
So how can you go from these 10 million customer reviews?
40:07
To actionable. Insights to say, okay, these are
40:10
like the top five, complains our customers have.
40:13
These are the top 10 feature requests we have so, so that you
40:17
can really have these insights. And an interesting direction, is
40:22
the combination of text understanding was generative
40:25
models. So, say, if I asked whatever
40:28
generate models you using like, hey what other top feature
40:32
request users of my company have The model maybe makes up
40:38
something. Where says, sorry.
40:39
I can't answer this because it doesn't have like any insights
40:42
on your company and what the 10 million users want.
40:46
And, and here in interesting directions, really you plug it
40:49
into your pocket, disgorge to Twitter to, I don't know.
40:53
Some survey you run and say, okay, here's like everything.
40:56
My users are chatting about talking about one of the top
41:01
features they do and how can we implement it?
41:04
And then you have first a search phase two, Over like a hundred
41:07
millions of messages, you have from your users in different
41:10
channels, narrow it down to say, okay, be solid top 10 features,
41:14
users request and an interested in, like, for example, improve
41:19
the latency of the model and then the generative model says
41:22
yet. So top requested feature from
41:25
the users, improve the latency of the model, there are like
41:29
different ways, how you can improve the latency of the model
41:31
faster Hardware. You just took distillation
41:34
quantization of models remand. So sorry, you have a really
41:38
grounded in some facts and some knowledge.
41:41
And are you just so so paints, a bit more of a clear picture?
41:45
Are you old exposing, an API, and expecting people to build on
41:49
top of cohere or you planning on building everything yourselves.
41:55
So, we're providing these models Aston API.
41:59
So there's like one side has an API, you get an API key.
42:02
You can request it say, okay. There's like a search model.
42:06
Then you can be like a really nice search function across many
42:09
languages or generative model. You given the prompt you get the
42:14
output. The other option is like on-prem
42:17
deployment. So currently we're on a wsh
42:19
maker where you can take the model hosted in your own
42:22
environment was connected to data privacy.
42:26
So there's like no risk. That any sensitive data is going
42:29
away from your Cloud environment.
42:32
And we're adding, like, more and more capabilities, like really
42:36
teach them all. How can my search and recent
42:38
over? I'd say your 10 million customer
42:40
reviews but obviously, you still need to.
42:44
It's a tool for developers, so you need to bring it to still
42:47
into into your product C. Okay.
42:50
Who do your Terrace? Is it like a B2B b2c and what do
42:55
you need to build myself? So it's a building block for you
42:58
as a developer. Yeah, makes sense.
43:00
So correct. Me if I'm wrong because I'm had
43:02
this use case in my mind for the mo UPS community that I've been
43:06
wanting to do for a while. And I think we'll probably do it
43:08
with some kind of hackathon later on down the line.
43:11
But there's a lot of podcasts out there of the ml, Ops
43:15
community that we've put out, and potentially, we transcribe
43:18
all of those podcasts, and then we can search them.
43:21
And we can have that indexable ability, and there's a lot of
43:26
conversations and great threads that have happened in the
43:31
Community slack. So, potentially with cohere we
43:35
would be able to plug that into our slack or we would feed all
43:40
of our data to cohere and then it would give us those insights
43:45
that you're talking about and we would be able to search and ask
43:47
things of it. Yes.
43:49
So for search technically called Works, they like two models one
43:54
so-called embedding or vector model.
43:57
So it's takes the text Madness dance, the content of Of the
44:01
text and transforms it into a representation that's
44:04
understandable for a computer, which was like a vector.
44:08
And that's what you come by and we'll select a database like vv8
44:11
Pinecone Kieran. So you say, you take all the
44:14
podcasts of scripture. Get the representation that's
44:17
understandable for the computer, and put it into into your vector
44:21
database, and then you already can build, like first valuable
44:25
thing, like a search bar as you note from Google, you can type
44:28
in and say, when did they talk about?
44:31
That's security and that's language models and that it will
44:34
pop off the ride podcast. Say, okay here minute 15 to 20.
44:37
They are talking about six year to year nationally, which
44:40
months. And then the next step, it's
44:44
still in really early stage research is like plug it into a
44:48
generative model so that you can reason about like in Texas.
44:54
So for example, Big Challenge is identifying personal information
44:59
in text so you could have It's like, okay, when I like ways to
45:03
identify personal information in text and how can I remove it in
45:06
Python, and then you have the search component, find the
45:10
podcast, talking about the security issue.
45:13
And then the generative model, takes the ideas in Christ, like
45:16
a python code for you to remove security card.
45:19
Test credit card numbers and Social Security numbers and
45:22
email addresses and phone numbers from your data.
45:25
So that's that's the future go. Where want to go so far?
45:29
You get the search results which can Also be really cool for for
45:33
the audience to say. Okay that's that's an
45:35
interesting podcast. I want to listening.
45:38
Yes. I think there's so many
45:40
potential avenues that we can hack around on with all of the
45:45
content that the envelopes Community itself has produced
45:49
and the ability with these large language models to do stuff on
45:53
top of it or gain insights from the different stuff.
45:56
So so big, big his case by convention is also like we're
45:59
going to launch soon sucker. Only it's closed beta soon, in
46:04
an open Beta is like, aggregation and only source of
46:07
results. So, what you could take as,
46:09
like, all the questions you get in Discord slack, analyze it and
46:14
say, okay, these are like the top five questions.
46:16
People are asking all the time, and then you can think about
46:19
like, okay, let's do a podcast on this.
46:22
So you see a lot of people ask about, I don't know, AI for
46:26
other languages. And you you say, okay.
46:30
That's, that's interesting. Apparently, there's a lot of
46:32
interest in this and all of our different channels.
46:35
Let's find great speakers talking about how to do AI other
46:38
than English. Yeah, I love that that would be
46:41
very useful and that's almost half news networks aren't
46:45
working already which is what they do.
46:48
Is eat people. Usually scar information on
46:52
Twitter or other platform for opinions.
46:55
People have about a certain topic and then they'll bring on
46:58
An expert in take those opinions and challenge those opinions
47:01
from like all sorts of angles with the experts?
47:04
Yeah, so when it comes to the work that you're doing that go
47:09
here, one thing that I think about is their open AI, which is
47:15
probably like it's sucked all of the air out of the room and
47:19
everybody knows and has played around with large language,
47:23
models, probably because of the chat GPT phenomenon that
47:28
happened a few months ago and they have a very strict policy
47:33
on their models, being closed source.
47:37
And then there are some open source versions that are trying
47:40
to come out and have come out like the notably like G PT J.
47:45
Or we have I think recently together just open source like
47:52
chat kit or something. And remember the name exam I'll
47:55
and type higher isos open source community.
47:58
Raising in terms of names. Yeah, exactly.
48:03
And so I'm wondering, like, how have you thought about that?
48:08
Do you talk about that or is that something that you don't
48:11
even enter in those discussions at cohere?
48:13
Yeah, of course. And well no I mean I hope before
48:16
cookie I work for having Face, which was the central hub for
48:20
open source models in the space, it's an interesting discussion
48:26
people have in the community. First, in terms of capabilities,
48:31
so open source models, currently lack behind compared to to like
48:37
a GPT or to go here or in traffic or AI 21.
48:41
Yeah. Also, because open source has a
48:45
challenge, who's developing it? It's all from like academics was
48:53
like some butchered open source, individual maintainers for like
48:56
Reddit really dedicated but it's Really hard to get like
48:59
consistent dedication in the space and also the requirements
49:05
for these molds, I think in the future, they become bigger and
49:08
bigger. So we're teaching them how to
49:11
use search, but soon, they will also be able to use tools.
49:16
So like some simple tools, like a calculator and be able to plug
49:22
in like to your calendar like moving meetings around and
49:25
communicate with people. I don't know.
49:28
Check flight information, check the weather's.
49:30
So when I ask like, hey I do next week to Toronto Canada what
49:34
should I pack it will first check the weather for Toronto
49:37
next week. Say okay.
49:39
Next week it will be snowing or the sun will be shining and
49:42
based on that. So I give you a recommendation.
49:44
What should you packing your suitcase?
49:46
And I think that'll just put it in the suitcase directly.
49:49
Yeah, and that's the ultimate goal, right?
49:53
And that's I think it's challenge.
49:56
Fall open source tool. Keep up and maintain it and
50:01
developed it s. Interesting discussion is also
50:05
security like like, if when the model is out, its out for
50:11
everyone which is on the one side, great and positive because
50:15
everyone can use it without restrictions and without the
50:17
gatekeeper. But also, what you what, we
50:20
discussed previously, the bad guys also can use it and there's
50:24
like no one in no way to prevent them too.
50:28
Train the models to create like a really compelling hate speech
50:32
and then use them at scale on Facebook, and Twitter for the
50:36
next us election and produce. Like millions and millions of
50:40
political motivated, tweets, post, whatever.
50:43
And flood, every aspect we have That's not really interesting
50:49
discussion mainly because I don't remember if it was
50:52
Microsoft or Google that released one of the mothers
50:56
ever, so close water, which is a hard research exists only if you
50:59
remember and then somebody put it on BitTorrent and give the
51:03
torrent link to everybody. Yeah.
51:04
For like the hyperparameters and everything, everything.
51:08
Think I'm water was available to everybody.
51:12
One of the, one of the things I've been hearing from, a lot of
51:16
people is There will be War for can open source as it comes when
51:20
it comes to building more. Sophisticated models by the
51:23
bigger companies and open sourcing them.
51:25
And essentially after that attack or I don't know if you
51:29
call that an attack because every where's Byam a teenager,
51:32
it was done by one of the maintainers after actual model
51:36
themselves and then Facebook put out there.
51:38
Llama model in public. Do you think we are going
51:42
towards that direction? Very will see the bigger
51:45
companies pouring there. Models are there for everybody
51:49
to be able to exist them and if so, how do we maintain Security
51:53
in those things or do you feel like Vicki's again?
51:58
Thank you and do some sort of Competitive Edge unless you're
52:01
training your models? Because again there's a question
52:04
of the models going steel after like a certain point in time do
52:09
you see more companies going towards the open the I Direction
52:12
where they have said after GPT for witches, you know, we're not
52:15
going to release the model. 100 pages.
52:18
But if you don't, I'm going to give you any information in
52:21
those hundred pages. So yeah, I'm a big fan of the
52:26
open source Community heavily contributed, but I'm afraid it's
52:30
going more to like knost apis. On the one side is the cost
52:36
like, your for these foundational levels.
52:40
It's like tens of millions, so it's like a really.
52:44
I mean, it's really generous that Google has been putting out
52:47
so many great models like Purge T5 but it's like yeah you need
52:52
to think about like you invest as a company like 50 million in
52:56
the development of a new model. Is it reasonable to put it out
53:01
for free for everyone? So, that's, that's where you
53:03
need like a really good income stream from ads on Google,
53:07
search to be able to do this. But for many other since I'm
53:10
really hard to do this. So, so what do you see?
53:14
Sadly. A lot of these newer models from
53:16
the big companies. They I'm like, not really open
53:19
sourced and more so so Google press the completely reduced and
53:23
stopped open source models. So all the interesting new
53:26
Google models pongee, Chiller retro, They are not publicly
53:30
available and Microsoft, is also kind of reducing it.
53:34
Facebook kind of still does it, but western singer, it's like
53:40
non-commercial restrictive only for researchers.
53:45
I mean. Yeah, there was leakage on
53:46
BitTorrent and then set-asides, it's a challenge, the others
53:52
like legal and misuse of the models.
53:55
Like, for example, stability, a I was stable Sunsets was really
54:01
interesting, really put a lot of good things in the community but
54:05
as a company they have to find like legal claims in terms of
54:08
copyright infringement and also, sadly stable diffusion or so.
54:14
Long used fold off bad things to my crepe pornography of like
54:20
underage people and like has dedicated models for that of
54:25
ex-partners where you create like these photorealistic images
54:29
To put people into like really up seeing some situations and
54:34
pictures and then spreading this on social media and say, hey,
54:37
here's I don't know. Joe Biden, Donald Trump or
54:40
whoever runs for president in like really bizarre scene and
54:44
then spread this misinformation. And this is like also goes back
54:48
to the company. Say, Hey, you put out this model
54:51
which has been used for all these bad things.
54:55
And so it will be interesting from from like a governmental
54:58
perspective, regulation to see. Okay, what was the boundary is
55:02
the company responsible for the misuse?
55:05
And this can be typically better protected in like no closed API
55:10
where you can say, okay it's not possible to create like any
55:14
pornographic material so you cannot spread around my pictures
55:18
from ex-partners. Ham with these tools and that
55:23
was no open source because they are people can do anything they
55:26
want. So it's the Idea then, do you
55:30
feel like the world that we will be living in is going to be a
55:36
world of apis? And is that the main Revenue
55:42
generator you feel especially when it comes to just looking at
55:47
cohere and you say, all right, we're gonna go all in on the API
55:51
and hope that that is what as you mentioned before will
55:56
sustain us. I mean Google lucky enough, they
55:58
have a cat Cash Cow in a different part of their
56:00
business, but for cohere, it seems like you have to figure
56:04
out a way to make that API work, right?
56:07
So, so sad that we don't have the cash cow like Facebook
56:11
Oracle, go forward, not yet, what?
56:13
Be nice and open up a lot more opportunities to do research and
56:18
open source. And but yet in general, I mean,
56:21
in technology there are a lot of great Technologies which are
56:25
closed source which were using every day when we look at S like
56:30
genomic DB. It's not like an open source
56:33
around database, easy to F 3 and so on, it's like everything is
56:39
closed Force by, we're really comfortable to see it as a tool.
56:44
So you build a application web application that's needs to
56:47
scale at a global scale. So you use AWS to normal DB + F3
56:52
and ec2 and you're happy to use it as a developer.
56:56
So so I don't think that there's, like, a lot of Demands
57:00
to say, hey, hey W as we expect everything to open source and
57:04
give all the tools you have Versa.
57:07
Yeah, it's an API, I pay for it. I pay a premium for it, but for
57:12
many use cases, it's a great cloud provider and I mean, it's
57:17
really hard to replicate it with some open source free versions.
57:21
Or on the flip side, it's actually a WS that will come and
57:25
grab the open source stuff and then say now you got to pay for
57:29
it. Yeah.
57:30
I mean with what's elasticsearch wants to interesting dispute,
57:35
taking the open source elasticsearch make a managed
57:38
service audit out of it and shattered for it.
57:41
And I mean that that's in general challenge for
57:44
open-source developers maintainers and how can you
57:49
still crying like a nice and it keep the intent and then try to
57:55
to avoid, like the misuse Yeah, if you're like, at some point,
58:00
you have to pay your developers because they have to live, they
58:02
have to pay rent. They have to buy food.
58:05
Who is uh, so if you really want to build something big, you have
58:09
to find like a way to make it substantial.
58:11
Like make make a living with it. All right, everyone, I just want
58:16
to jump in real fast because I have a huge announcement.
58:19
We will be having a part two of our large language models in
58:23
production. The first conference was such a
58:26
success that we had to follow it up because there were so many
58:29
incredible speakers that reached out after the first one.
58:32
I couldn't help myself. So it will be on June 15.
58:35
And it is going to be online. So anyone from anywhere can
58:40
join, but if you thought the first one was good, then pulley,
58:43
are you in for a treat? We have some confirmed speakers
58:46
already. I'll just go through a list of
58:48
them backed, a from date of bricks, you know, the founder of
58:51
the heat of Rick's and Spark and all that good stuff from
58:54
Recently, just had him on the Pod.
58:56
He's coming back to Keynote. You've got chip, Yuan, who is an
59:00
amazing person and incredible follow on social sharing.
59:05
So many insights and has been dropping knowledge bombs when it
59:08
comes to large language models recently you've also got Scott
59:12
McKay from MIM dot AI. You've got the Deep speed team's
59:17
gonna be there from Microsoft that is pretty epic and Sahar
59:22
more from Strife. Another stripe alumni is
59:25
Emmanuel Emison, he's formally of stripe.
59:28
Now he's working at one of these large language model companies
59:31
that you've definitely heard of, but I don't think I can feel the
59:33
beams on yet, and And we've got Lance Barton in as from number
59:38
station. George Matthews from inside
59:41
Partners. This is just a glimpse at the
59:43
whole list, go sign up. You've got all the details in
59:47
the link in the description. Huge shout out to the sponsors
59:50
redis for showing us some love and making sure that this event
59:54
happens. Our guila any scale continual
59:57
chroma. Thank you all, and you can
1:00:00
expect all kinds of bad musical improvisation.
1:00:04
The brakes like I know Normally do it.
1:00:06
And who knows? I may make that I hallucinate
1:00:09
more than chat GP t-shirt available for sale again.
1:00:13
Check it all out. Links in the description will
1:00:15
see soon as you're looking around and you're you're seeing
1:00:20
all the other players that are in this space and it seems like
1:00:24
every day there's new ones. And they're kind of converging
1:00:28
on different use cases and things that you hope to
1:00:31
accomplish with the large language models.
1:00:35
What are you hopeful about with the direction that coheres going
1:00:40
in? That makes you think?
1:00:42
All right, we're distinguishing ourselves because of x y z as
1:00:46
opposed to all of these models being more or less commoditized.
1:00:51
Yeah two things. Oh it's so open the eye and as
1:00:55
your own Pay. They try to use it to get
1:00:58
customers locked into like Azure, my opinion said, okay,
1:01:02
you have to bring your data to measure so that Cricket.
1:01:05
We're doing different route and say, okay, we want to bring all
1:01:08
models to you, where the data is, we already like on Google
1:01:12
Cloud but also on AWS can run in your own environment while also
1:01:17
plan to, to expand it more and give them more capabilities to
1:01:21
customize. So that you say, hey, I take the
1:01:24
model and I can already cost Is it for my use case?
1:01:27
How should I write? How should you respond?
1:01:29
One of the best tools to talk adopted second big differentiate
1:01:35
has a strong focus on text understanding so we have
1:01:38
extremely good tools that allow you to do search to do text,
1:01:42
classification to do like the aggregation so where you can
1:01:46
create a new value for users already now because you are able
1:01:49
to search on your podcast and aggregate questions to see what
1:01:52
I like, the most common questions to plan for the next
1:01:55
podcast. But also to really marry this
1:01:58
text understanding with the generate model.
1:02:01
So it really understands on your inbox on your CRM data on your.
1:02:08
I don't know on your podcast, what have you been talking
1:02:11
about? So, if you ask of the model,
1:02:13
hey, write a follow-up email to an EOS.
1:02:16
Thank him for the podcast. We made that can go to the
1:02:20
transcript of the podcast. Understand what have papers
1:02:22
discounts go to all previous emails, take it into account and
1:02:26
then we create a personalized response and text for that.
1:02:31
I'll dude, I can't wait for that.
1:02:33
I mean, the hard part is that sometimes you just get blocked
1:02:37
by the other walled Gardens, like, I know that we've chatted
1:02:41
a lot on LinkedIn. And so if I wanted to do that,
1:02:44
let's imagine that everything was there.
1:02:47
I couldn't do that because of the LinkedIn apis.
1:02:49
It wouldn't allow me to plug into their messages, right?
1:02:53
Yeah. That's, that's a challenge.
1:02:54
I mean, we have so many Dyke data, sidles, you have data as
1:02:58
and I don't know, Gmail and your private email on To end on
1:03:02
Twitter on slack. I'm just porch notion Ocean.
1:03:07
Think You're Going forwards. Big, big push forwards.
1:03:12
Like how can you combine it? And they aren't really
1:03:14
interested in the field or I think players in the field can
1:03:18
do it. I really well positioned.
1:03:20
So I think Microsoft's well-positioned.
1:03:23
They have like teams they have. Yeah like like Outlook 365 so
1:03:30
that's a lot easier. When you have like all your
1:03:32
communication and teams, you have all your communicate while
1:03:35
the transcript and teams, you have Outlook and Microsoft
1:03:39
Office. Google is also wet position with
1:03:42
Gmail and Google Docs and Google meet and so on.
1:03:46
So I think for these players yeah like access to data will
1:03:50
become critical and then yeah once we figure out, how can the
1:03:54
generative model access all your data in oil?
1:03:58
Yeah, you know, you come like the whole Corporation and you
1:04:01
can ask questions. Questions.
1:04:02
Like okay. One other, the top feature
1:04:04
request by my users, checks your slack teams, whatever disco
1:04:09
Channel analyze it. And then I was like, hey Craig
1:04:12
like a project plan to implement this feature.
1:04:14
How can we improve? I don't know.
1:04:16
Latency and then so gives you like a really compelling project
1:04:21
plan on it. You can already check with the
1:04:22
database. Okay, who would be like good
1:04:24
people who worked on such things before?
1:04:28
And I know already proposed, like, okay, this would be like
1:04:30
the teams and people I would need and maybe where do we need
1:04:35
to update the database or where do we need to update our code?
1:04:39
What? Yeah, Big Challenge and
1:04:41
Staffing's reception as well. So it's a big Challenge and
1:04:45
Staffing as you Steph while you, you talk to the people, you
1:04:49
know. But in a corporation there like
1:04:51
many more people like in Google, you have like I don't know,
1:04:54
something like 200,000 people make.
1:04:57
K Engineers back. Who's the best person to
1:05:01
optimize? I don't know half where I keep
1:05:03
EU instructions and and you don't know it.
1:05:06
He just know the 102 co-workers maybe but you couldn't look at
1:05:10
previous plans, projects documentation and say hey this
1:05:13
random dude from some different country has a lot of expertise
1:05:17
and worked on the love similar projects and I have proposed
1:05:20
this person should also enjoying this new project.
1:05:23
Interesting. And this makes me think Maybe
1:05:28
it's probably pay our last question, but wrapping up what I
1:05:32
wanted to discuss with their two kinds of companies that we're
1:05:35
seeing in the space right now. One are the product companies
1:05:39
that are building on top of these apis one other companies
1:05:42
that are building these Foundation models.
1:05:45
And while there are some smaller starch, if you still can't go
1:05:49
here and you don't come as startups as compared to, you
1:05:53
know, like here Facebook Google which are almost like a to do.
1:05:57
Eleven itself if we still call them fungi from.
1:06:00
Not sure when their academics anyway.
1:06:03
Let a, which what I do focused as a person, what are you more
1:06:10
focused towards and what are you reading or so these days?
1:06:13
Are you reading more stuff around?
1:06:15
Foundational Waters themselves around, semantic, search around
1:06:20
yours, or spaces, or are you reading Morse stuff around?
1:06:24
The product set the product spaces?
1:06:27
Or, you know, one of the apps that can be built on top of
1:06:30
this, you know, like whirring Twitter in seeing what a people
1:06:33
ought to be building with queer, then secondly, spanning in
1:06:39
general, a lot of time with users and listen to users.
1:06:43
So a lot of Engineers, they fall in the Trap, especially on the
1:06:47
machine Learning Community to optimize certain matrices, which
1:06:51
are not that governmental. So you have some Benchmark and
1:06:54
you try to push the QC and you think, okay, if you go from ATI
1:06:58
8.2 accuracy to 88.5 accuracy that this will make like a big
1:07:03
impact with so users. But users State they often have
1:07:08
like other needs and requirements and challenges.
1:07:10
So they don't really care about like these points, very
1:07:13
curiously difference, rcml model and they have Community
1:07:18
different problems. So one thing in semantic search
1:07:23
right now, a lot of people face a challenge, how to chunk text.
1:07:28
So far semantic search. Only works on my short text like
1:07:32
a couple hundred tokens. If you have longer text, you
1:07:36
have to segment it to like smaller chunks of 100 tokens
1:07:40
each and like compute, the vector of the community
1:07:43
representation for each of these chunks and yes sir.
1:07:47
So many people that faced the challenge.
1:07:49
Like Hey how do I do the segmentation I have this work
1:07:52
document PDF document. This clinical note this patent
1:07:56
text is earning report what's like a nice chunking?
1:08:00
How can I take these two hundred Pages running reports from
1:08:03
Johnson &? And was tables and images and
1:08:07
sections and subsections and titles and footnotes and had
1:08:11
nose and change it into like a nice representation to input as
1:08:15
to the model. And then they do not really care
1:08:18
about these point to difference on some Benchmark for they say
1:08:22
okay. Yeah I chose it because it's
1:08:24
really nice and I don't have to worry about jumping for example.
1:08:29
So so often you'll find when you talk with people that other
1:08:32
things are more Relevant than really these leaderboard chasing
1:08:37
and in try to to be the top on the kale competition or
1:08:41
whatever. Public academic leader boards.
1:08:45
Are you trying to chase? Oh yeah.
1:08:49
We've been seeing that since 2020, and thereby laughs
1:08:53
Community. Not many people in that
1:08:55
Community actually care about care.
1:08:58
Go as much as how am I going to either save my company money or
1:09:04
make my company money. Yeah that's yeah.
1:09:07
I mean at the end you want to use machine learning for these
1:09:11
to make money while Safe Money. And that's yeah, often different
1:09:16
things around. Yes, and it's and it's really
1:09:19
misaligned and also bringing back why switch from Academia.
1:09:22
So I could, he miss really many case, this leaderboard shaysing
1:09:26
creating systems were not practical.
1:09:30
So they are like when in search there are some systems that
1:09:34
takes like a minute or so to for single search query and cost
1:09:38
like a lot of money to do. And I mean I would not use
1:09:42
Google if it takes like a minute to give me life the top 10
1:09:45
results even when it's like, Maybe it's better and 1 out of
1:09:49
100 1 out of 1,000 queries. Let's say no, please give me
1:09:54
something that's fast. I can type in, I don't know, I
1:09:56
type in YouTube and I want to click on YouTube because I'm too
1:09:59
lazy to remember youtube.com. Probably.
1:10:02
I mean, that's the most common. Search query on long run when
1:10:06
Google and says. So that's where it's find that
1:10:09
the industry setting interest more interesting, cause it takes
1:10:13
more Dimensions into account. Not only likely towards Jason
1:10:16
Kegel, They're like an impractical system.
1:10:19
That's really good. On this arbitrary Benchmark, but
1:10:21
build a systems people off by build a product, people are and
1:10:25
find helpful for that daily work.
1:10:28
I love that product mindset. So you guys got anything.
1:10:31
You want to promote or mention coming out of cohere any product
1:10:37
or you all hiring anything you want to give a shout-out to?
1:10:41
Yes. A big project when a giant says.
1:10:43
So I'm originally from Germany and Germans, my native language
1:10:48
and a lot of focus has been in the field in search in English.
1:10:52
So on English we have like amazing models for put for any
1:10:56
other language. Semantic, search Savvy, didn't
1:10:59
work and so big project. First project I've been working
1:11:03
on. It could hear was like the
1:11:04
multilingual text understanding model, which works amazingly
1:11:08
well in like multilingual and also Crossing we'll search.
1:11:11
So I can phrase a search query in French and find like a
1:11:15
Chinese document giving me the answer on that without any
1:11:20
translation which is like super interesting.
1:11:23
And in Europe, you have many of these multinational companies
1:11:27
and then Colleagues and Friends Day ride and document in friends
1:11:31
and colleagues in Germany and German and French and Spain and
1:11:34
Spanish. So a lot of information is lost
1:11:37
due to the language boundary and now it's accessible when the one
1:11:40
side for for search for the other is like text understanding
1:11:44
classification. So so hate speech to text him as
1:11:48
a big challenge in online gaming so you put on like your team of
1:11:52
five. He put out team which was like
1:11:54
really successful get a lot of traction.
1:11:57
And the App Store, you get a lot of users using it.
1:12:00
But sadly people spit out like unfriendly words.
1:12:04
Hate speech in like many, many different languages and it's
1:12:07
like really hard to like really do content moderation in so many
1:12:11
languages and ask them all understands.
1:12:13
All the languages is now sufficient to provide like 10
1:12:17
examples of hate speech in English.
1:12:21
And then the mall says he, okay, I understand you want to
1:12:24
identify hate speech unwanted comments.
1:12:28
Be able to transfer. It's like hundred languages and
1:12:30
then you got like really quick, nice solution to really
1:12:33
understand. And work.
1:12:34
In this multilingual, cross-linked will setting.
1:12:37
And this is at the risk of continuing this conversation if
1:12:41
the, since you jumped into that. I mean how much of your time or
1:12:46
the team's time is spent red seeming, the models to make sure
1:12:50
that you do have that capability.
1:12:54
So so was red. Teaming.
1:12:55
You mean like try to do like adversarial attacks and try to
1:13:00
Yeah. And try to basically jailbreak
1:13:03
it because people said I am so this is more challenge for for
1:13:09
text on a text generation. I'm supervising text
1:13:13
understanding. So here we are in our really we
1:13:16
say, yeah, we're we're not really holding back his speech
1:13:20
from the training of the model, because many want to analyze
1:13:24
hate speech. So you want to go into a user
1:13:26
comments and filled her hate speech says.
1:13:28
So we really want them all the time.
1:13:30
Understand it, but because the model is not producing any
1:13:33
content, so it's not able really to produce hate speech.
1:13:37
So just says, here's a text, I put it into like the area with
1:13:41
hate speech because it's like, racist or sexist or whatever,
1:13:45
and then you can take the action on it and say, okay, please.
1:13:48
I don't want to highlight racist, sexist, content on my
1:13:52
platform. So, so far us red.
1:13:55
Teaming, assist not bad, big of a challenge, but for the
1:13:58
generator model, Yeah, it's a big, big challenge for the team.
1:14:04
Awesome news, dude. I look forward to the day when
1:14:07
we get to catch up in Frankfurt or Wiesbaden or wherever maybe
1:14:14
maybe in Berlin for having a conference in September.
1:14:18
I think I may have mentioned it to you but we should get you to
1:14:22
come and stop by and do a talk or something.
1:14:24
Well, I've been trying to get you to give a talk in Frankfort
1:14:27
at the local, meetups to that'll be awesome.
1:14:30
This was really cool and thank you for your time and chatting
1:14:33
with those about all this stuff, great likewise, really enjoyed
1:14:36
to be here. And yeah, looking forward.
1:14:38
I'll AI features, developing so quickly and we'll see what we
1:14:43
see this year. Fooled ya right on.