Latest / Agentic Conversations (formally mlops.community) / [EXCLUSIVE EPISODE!] LLM Key Results

Transcript

[EXCLUSIVE EPISODE!] LLM Key Results · Agentic Conversations (formally mlops.community)

0:14
What's going on, everyone? I am Demetrios your host and
0:17
misses the envelopes Community podcast.
0:19
Today, we have a very special edition of this podcast, if you
0:24
couldn't have guessed it by the title or ready, we are going
0:27
over the survey results from the Large language models in
0:31
production survey that we put out.
0:33
We are going to be reviewing eight key insights.
0:36
From the large language models surveys and just in case you
0:41
want to go through and check out the data on your own.
0:44
It's all out there. The link is in the description.
0:47
It's all open source. So don't take my word on any of
0:50
this. This is just a summary of some
0:53
of the key findings that I found, as I was wrestling with
0:56
the data, it is all unstructured data.
0:59
That's I learned it's not the easiest to do but now I know and
1:04
for the next one I'm going to make sure to give it a few more.
1:08
Multiple choice questions. We plan to do another one of
1:12
these at the conference that is happening on Thursday, April
1:16
13th. So that is a virtual conference.
1:19
It is free and we are going to release all of the data again.
1:23
And another report this actually came from me, putting together
1:29
the report and I I thought, wow, you know what would be awesome
1:32
for those people that love podcasts like myself.
1:35
If I just went through some of these findings that I am putting
1:38
together in the report and I talked about them on this
1:41
podcast, Last thing before we jump into all of the key
1:48
insights and great findings from this.
1:50
If you find any value in this episode whatsoever, the highest
1:54
leverage thing that you can do right now is share this with one
1:58
other person. You think will like it?
2:00
It will help us grow. If you think about how you found
2:03
this podcast, I imagine it's someone that told you about it.
2:07
I don't think it was my social media Antics that got you to
2:11
turn in. So Feel free to share this with
2:14
one other person that would appreciate these key insights
2:19
from the large language model in production survey that we put
2:22
out. Alright, without further Ado,
2:25
let's get into it. Okay, let's start by who
2:40
participated in this survey. We want to know the profiles of
2:44
the people we want to also know how they found out about it.
2:47
And again how many people participated and from what dates
2:53
was this survey open, we saw a large number of The Usual
2:58
Suspects from the ml Ops community and there was a
3:02
varying degree of seniority. But for the most part, I was
3:06
quite surprised at how How many senior level participants we had
3:11
that were filling out? This survey, lots of different
3:15
names for the titles that each person had but at the end of the
3:21
day, I was able to bucket them into a few different titles that
3:25
I think encapsulate, the personas who participated.
3:31
So these different buckets were machine, learning, engineer, /ml
3:36
Ops engineer. I don't really know what.
3:38
Differences there. But obviously, some people have
3:41
different titles and that's what we're going to go with.
3:43
Then there was Data scientists of all degrees, different kinds
3:48
of scientists. You even had the outlier of the
3:51
deep learning scientists, all of that, great stuff.
3:54
We also had head of data, head of ml, head of ml, Ops cdo's
4:01
those head of positions. There was a quite a few of those
4:05
and I'll break down all the percentages.
4:08
Has in just a moment. We had some founder CEOs,
4:11
actually a bit of founder, CEOs, some ctOS product head of
4:16
product product owners and then miscellaneous was the rest that
4:22
were those outliers. Like there was one analytics
4:24
engineer. There was a few Consultants, a
4:26
cloud engineer marketing manager, at that kind of thing.
4:30
For the machine learning Engineers.
4:32
We had about 22 percent of the respondents being machine
4:37
learning engineer. Again, this is mainly senior or
4:41
staff machine learning Engineers, but there was also
4:45
just the normal machine learning engineer.
4:47
We had data scientists with varying degrees of seniority,
4:51
but the majority were those senior data scientists and there
4:59
was a little less than twenty percent of the respondents being
5:04
data, scientists, or identifying as data scientists, we have Had
5:08
about 13 percent being the head of data roles and a little over.
5:15
15 percent were the founder CEO, roles under 5% were ctOS, just
5:22
at five percent was the product owners and then the
5:28
miscellaneous category got about 15 percent.
5:32
In total, we had 108 people fill this out, which for me, Awesome.
5:38
I think we can do better the next time but that's incredible
5:41
that we got 108 people to talk about how they are using large
5:45
language models in production. The way that we got the word out
5:48
about this was about 20 minutes before we had our Roundtable
5:52
session on using large language models in production that
5:56
happened to almost a month ago. We asked the attendees to fill
6:01
out this survey and we also had a few of our influencer friends,
6:06
tweet out a link to the survey. To get a bit more traction.
6:09
The survey was open from March 11th until around March 23rd.
6:15
That's when the majority of the responses came in and we started
6:20
playing with the data. Okay?
6:22
So let's take a look at this actual survey and what some of
6:26
these key insights were then I'll break down how I'm going to
6:29
structure the rest of this episode.
6:32
The main Insight Above All, Is that the majority of companies
6:38
Are still entering the early stages of using large language
6:41
models in production in their business, or to put it another
6:47
way. It's we're still figuring out
6:50
how we can use these large language models and what use
6:54
cases they're going to produce the highest value.
6:59
There are some challenges right now.
7:02
We saw a lot of respondents saying that there's challenges
7:05
in cost in latency and The output variability and other
7:11
infrastructure sophistication efforts, still hinder teams from
7:16
extracting the most value from their Investments.
7:19
So as a result of this, you know what's coming, there's going to
7:23
be an influx of venture capital jumping into this sector.
7:27
It's already happening. You probably have seen some
7:31
developer tools for large language models getting funding
7:35
recently in the last couple weeks.
7:37
Well, Please see a boom in the number of LM infrastructure
7:41
companies and developer tooling providing services to overcome
7:45
these obstacles that we found in the survey.
7:49
I want to highlight eight key findings right now and then I
7:53
want to structure the rest of this podcast into four different
7:58
pieces. One is the different use cases
8:01
that people talked about. The next is, the special tooling
8:06
that people were saying they They were creating for this new
8:10
paradigm, then there are the challenges and concerns that our
8:16
respondents addressed, and the last one open questions and
8:20
opportunities. Let's dive into these eight key
8:24
findings real fast, and what they say about the llm in
8:29
production landscape. And just as a heads up, I may
8:34
refer to LMS as Lems because it's a little bit easier to say
8:38
And I've heard that maybe catching on.
8:40
So in case you hear me say that don't get confused.
8:46
Okay, so these eight key findings are as followed number
8:49
one, common use cases like text generation and summarization.
8:54
Those are quite useful but the participants of this survey are
8:59
trying to go deeper and they're exploring ways to use large
9:03
language models for data enrichment with data labeling
9:06
augmentation. Synthetic data creation and
9:10
question generation for subject matter experts, or get into all
9:15
of that, a little bit more in our use cases section because I
9:18
find it fascinating these creative ways to use the large
9:23
language models. Number two, is that the decision
9:27
to use large language models within an organization is still
9:31
unclear due to the high cost and unknown Roi on this number.
9:38
Is that the challenges around limbs speed of inference?
9:42
Make certain use cases. Absolutely impossible.
9:45
They're completely off the table.
9:47
Number four is that the current infrastructure is not designed
9:53
for such massive models. So that means the current
9:56
developer tooling, it doesn't take into account these gigantic
10:01
models and this leads teams to implementing workarounds for
10:06
quick fixes which which we'll get into more in the challenges
10:10
section. Number five, the fifth key
10:13
Insight is that companies are building custom tooling to help
10:16
with pains around output validation and prompt tracking
10:21
that. Again, we're going to jump into
10:23
in the key challenges because the output validation seems to
10:28
be like a thorn in the side. Number six, following up on that
10:35
output validation is Hallucinations present a real
10:40
ethical and business concern. So much so that for some
10:46
companies, it just makes using large language models as
10:51
showstopper they cannot associate their brand with this
10:56
output. Number 7 is that there is a
11:00
larger question looming in the air and it is which part of the
11:05
current workflows that we have. Have today will be replaced
11:11
versus will be augmented. Leaders are trying to figure out
11:17
what can we completely use large language models for and what can
11:24
we augment our workers with large language models?
11:29
And the last one number eight, is that prompting is a very
11:33
valuable skill, but even the best prom Oops, cannot force the
11:39
model to give reliable output. Let's get into these different
11:46
sections. The use cases, special tooling
11:48
challenges and concerns, and the open questions and opportunities
11:52
for your information real fast. Out of the 108 people that
11:58
responded to the question if they are using large language
12:02
models in their organization 60 point seven percent of them
12:07
said, yes they are. And 39.3% said no they are not.
12:14
So, talking about the use cases, there are quickly emerging a set
12:21
of use cases that lend themselves.
12:23
Well, to this type of technology in its current state.
12:28
This is dated on April 10th. So potentially in two weeks open
12:34
a.i. comes out with a new release that completely destroys
12:38
all of this. But for now on April 10th we are
12:41
talking about certain ologies and certain use cases that are
12:47
more suited for the use of large language models than others.
12:53
The first one that was surprising to me was that
12:56
recommendation engines are being rethought, this instantly made
13:02
me wonder is are we going to do recommender systems differently
13:07
and at a high level? It's quite fascinating to think
13:11
about because recommender systems, generally and
13:15
traditionally are very much the for real-time systems.
13:20
And these large language models do not Not have that capability
13:24
right now. However, the respondents did
13:27
highlight that they are using large language models in their
13:31
recommender system, workflow to help create semantic Vector
13:35
embeddings as a feature to help with feature extraction and this
13:41
was the most popular answer helping with feature extraction.
13:45
So asking large language model which features are actually
13:48
useful here and the last one is helping with Recommendations.
13:54
However, this one to me and this is just my feeling, it is not
14:01
the respondents but this one feels like the most juvenile.
14:05
The helping with feature recommendations, I do not see
14:09
that one being as useful in this moment in time just because of
14:13
the latency requirements that you need the next use case that
14:18
we saw many respondents talking about were what I am calling.
14:23
A niche chatbots. So chatbots, we all know them,
14:28
they've pinned the bane of the internet since being introduced
14:32
in whenever 2010 around then and we all have been on the phone
14:38
with recording. And it says, please enter here,
14:41
or please say what you're looking for that is annoying and
14:46
that is a horrible user experience.
14:48
Just like, I feel up until now chatbots have been a horrible
14:52
user. Terrence I always try to get
14:54
somebody on the live chat as opposed to speaking with a
14:58
chatbot up until now. So chatbots all of a sudden with
15:04
chat GPT they exploded and it started to show that there is
15:09
lots of potential here and how the different participants are
15:14
using chatbots was actually a bit surprising.
15:20
So why do we call them Niche chatbots?
15:23
Well, basically they are specialized large language,
15:27
model, chat box on narrow domain topics.
15:33
These chat, Bots are being used for both external and internal
15:38
facing scenarios with the internal being the most
15:43
fascinating to me. So the internal Niche chat Bots
15:47
are showing promise When they are helping workers quickly
15:52
query diverse knowledge, bases, across the company and you can
15:56
imagine that this is only going to become more relevant as all
16:03
of this disparate data is in a million different places and if
16:08
you can connect up a large language model that sits on top
16:12
of it and understands all of it has been fine-tuned with it or
16:15
is able to quickly get that data that you are.
16:20
Looking for and give you an answer back then that will
16:24
continue to prove out its value. This is a predominant one with
16:31
the majority of respondents answering with the phrase
16:37
information retrieval that is the idea and that is the use
16:42
case. That people talked about
16:43
internal uses of information retrieval and an outlier case.
16:50
Case for these Niche chatbots full of potential is internal
16:56
facing chatbots to help power up.
16:58
Managers by having them quickly, understand context around
17:03
projects, get the right data that they need or answer
17:07
relevant questions that they may have around project details.
17:12
So that's the niche chatbots. The next one up is around
17:17
customer support customer support.
17:20
Again, as I was mentioning earlier, it's not a novel use
17:23
case. It is however, quickly being
17:27
proven out as one of the strongest use cases, aside from
17:32
these chatbots that we were just talking about, there are other
17:35
ways that limbs are being used with customer support, one being
17:41
identity matching. So, determining, if multiple
17:46
records on file match, the same real world identity.
17:50
T. Another one being analyzing
17:53
customer service. Responses that is awesome.
17:58
So, being able to say which customer service responses
18:03
actually have value, and allowing the customer service
18:08
reps who get those responses back to gather data and quickly,
18:14
understand what the customers felt was good or bad and the
18:19
last But one, which goes hand-in-hand with analyzing
18:22
customer service, responses is analyzing the customer feedback
18:26
forms, and not only analyzing customer feedback forms, but
18:32
understanding the sentiment, so sentiment analysis in the text,
18:36
and then also automating the writing of reports based on the
18:41
feedback data. Again, this goes back to that
18:44
larger theme of the summarization is a killer use
18:50
case. Next up, we have coding and just
18:54
about everybody has heard of GitHub copilot.
18:57
As you may know, GitHub copilot X is out and it is leveraging
19:02
GPT for coding is another strong use case that we saw come
19:06
through and it is more than just asking GPT for to create a
19:11
function for you. Some of the respondents
19:14
highlighted that they are using large language models to help
19:19
them analyze logs to help. Them with security,
19:23
vulnerabilities, root cause analysis.
19:27
There's lots of things that you can just throw into these models
19:30
and get help with. But the bigger theme that is a
19:34
significant one. Is that these models are being
19:38
used like a translation service if we could call it that some
19:43
use cases for translating or obviously going from one
19:47
programming language to another, I have something written in
19:50
Python and now I would like to Write it in Rust.
19:55
However, I am not sure how strong that use case holds up.
19:59
And I would love to hear more from people out there because I
20:03
just feel like the level of abstractions when you're writing
20:06
it, python versus writing it rust or going to be completely
20:10
different. But we need to do a whole nother
20:14
survey just on how developers are using large language models.
20:18
But anyway, I digress, the other Ation parts that people were
20:24
talking about wer with just natural language to sequel.
20:30
That is becoming one that is showing promise or translating
20:35
an Excel report into python code.
20:39
So this idea of translation I thought was fascinating and that
20:43
was one that you see a little bit.
20:46
But I'd love to see how participants are taking that
20:49
even further and the last one on our use case map is the data
20:57
enrichment. So you have synthetic data
21:01
generation, synthetic image generation, you also have data
21:05
annotation something like clip, interrogator, or Dare we mention
21:13
Google training Bart with GPT there.
21:17
You've got a little bit of data annotation happening, I'm not
21:20
sure. If people are super stoked to
21:24
talk about that one, but it happened.
21:25
So might as well say it, there's also this use case that I get
21:30
really excited about, which is question generation and so it's
21:35
knowing the right questions to ask at various stages of a task.
21:41
So when you are going, and let's say you're working your way
21:44
through a project, the large language model understands the
21:48
development and how the Act is going and it is able to ask you
21:52
if you've done XYZ or if you've thought about this or that that
21:57
use case is really incredible especially if it's able to feed
22:01
in all of your specific data for that project and your company
22:06
and then it can help you get better at completing projects or
22:11
thinking through different side effects, that your decisions may
22:17
have the other piece when it comes to question.
22:21
Is analyzing data and asking if a subject matter expert has
22:26
considered XYZ. So these on the data enrichment
22:29
side, I think are incredible especially when it comes to the
22:32
question generation and we all know the quality of the
22:36
questions cannot be understated. All right, let's move on to
22:42
special tooling now and who and how the different respondents
22:48
identified their All tooling that they have and what they
22:53
were doing to make sure that they can successfully.
22:57
Put these large language models into production.
23:01
So out of the 57 people that said yes they were using large
23:05
language models in production and remember this is 60 point
23:09
seven percent of the respondents 51 of those 57 said that they
23:17
were building some kind of tooling.
23:20
To support their use cases. And this 51 number is a little
23:26
bit deceiving because it was 51 people in varying degrees of
23:32
sophistication or building internal tooling.
23:35
So some just said, yes, building internal tooling.
23:38
They didn't give any more details.
23:40
Others said yes I'm creating a product around it so you can
23:45
guess this goes back to this main theme of there's going to
23:48
be a boom in Products that come out Very soon.
23:51
Others said, things like we're using hugging face inference and
23:55
points and a few internal things that we've built around that or
24:00
were using rasa for chatbots, or some said, yeah, we're using
24:04
open e. I and a couple of internal tools
24:07
to support that some mentioned that they're using database
24:10
connections. And then you have what I would
24:13
say is the different or the more advanced use cases where people
24:20
are saying We've deployed our own open source, large language
24:24
model, and then you have people saying that, yes, they have
24:27
their own tools for training setups as well as inference
24:31
setups or another respondents said internally.
24:33
We use Bert to create embeddings, then we train a
24:36
smaller model on customer data. And the last one I want to point
24:40
out is a respondent who said that they have a platform
24:43
internally to develop and run models in production called stat
24:47
fish. So, there we go.
24:49
That's the varying. Um, degrees of maturity in the
24:54
internal tooling that people have built.
24:56
And this is fascinating, there was an example of a full-stack
25:02
toolkit that came through and it was Lang chain on open a.i.
25:08
Playground with little chat GPT prompt layer the tool and we V8
25:14
the vector data base. So I thought that was awesome to
25:17
see that. There's like a mean stack or a
25:19
marine stack, that is emerging and We also have some tooling
25:23
that has been created around output validation and we're
25:29
starting to see more tools come out because of that, I think
25:33
there's an open source tool now that is called guardrails,
25:36
that's coming out that should help with that.
25:38
But this is a huge one output validation is you'll see in the
25:43
challenges section. It's a headache.
25:46
The other pieces are because these language models are so big
25:51
Some of the special tooling that has been created is a high scale
25:56
server. On top of kubernetes that is all
26:00
built in-house. There you have it.
26:01
It's the rainbow of different tooling and I imagine it will
26:07
not be long before we see an explosion of tools coming out.
26:11
So that's a perfect segue into the main challenges and concerns
26:15
section of this, the current infrastructure.
26:19
Let's just face it, it's probably Matic at best because
26:23
teams are having to implement workarounds and quick fixes.
26:27
This one came up so much, whether you're handling, very
26:31
large documents, or your handling, very large models, you
26:35
have. When it comes to the documents,
26:37
you have a input limit, you have to use Vector databases, you
26:41
have memory requirements. During the inference, all of
26:46
that around the infrastructure and just dealing with a lot more
26:50
data and a lot. Lot bigger models.
26:52
Now, this may not be as much of a problem if we see more open
26:58
source models come out onto the market, I imagine a lot of you
27:02
have seen what was done with llama and how small they were
27:06
able to make that model. Potentially that could fix some
27:09
of these headaches. But I'm not quite sure around
27:13
the legalities of Lama and if you are able to use it for
27:17
commercial use, I should inform myself more about it.
27:20
But if anyone knows Just let us know in the comments and tell us
27:24
about what you can and can't do with Mama.
27:27
I just know there was some kind of drama or controversy around
27:29
it. Anyway, moving on, there is a
27:32
lot of pain around latency again.
27:36
What we talked about in those key findings is that certain use
27:40
cases just cannot be done. You can't do fraud detection
27:45
with large language models because the latency is a blocker
27:50
you can't wait around. For 20 seconds to determine if
27:53
someone is being fraudulent or not.
27:56
And why would you when the current machine learning models
28:00
in the way that people are doing fraud detection right now is
28:03
working quite well. Another huge concern that came
28:06
up quite a bit, was the cost concern and this was almost like
28:11
90% of the answers to the question around concern and not
28:17
only that. But if you dig a Little Deeper,
28:20
a big question is Does the high-cost actually have an Roi,
28:25
or are we just jumping onto some hype wave?
28:28
There's not really a queer way to show the ROI.
28:34
And so, that's what these respondents are trying to figure
28:38
out. How can we prove that this
28:40
actually is giving us a return on our investment, especially if
28:44
we are paying a ton of money to someone for the privilege?
28:51
Judge to use these large, language models, another
28:54
challenge, huge challenge, are the hallucinations and being
28:58
able to trust the output, just being able to reproduce the same
29:05
output and have quality output is easier said than done.
29:11
There are also ethical concerns around who is the nation's and
29:15
attaching your brand. To these hallucinations, we all
29:20
know what happened to Google. And the stock price after they
29:23
released Bart. And there was a bit of a
29:26
hallucination, I jokingly put that my New Year's resolution is
29:30
to be as confident as chat GPT when it just spews out bullshit.
29:35
I think that for any of these use cases that are high-risk
29:40
using large language models, is just a no-go because of the
29:43
hallucinations and the trust issues here.
29:46
You also have participants that mention explain ability is just
29:50
practically. Lee off the table and they're
29:52
trying to figure out ways to explain how the models have
29:56
gotten to the answers that they've gotten too but it is
30:00
increasingly more difficult and really getting to the ground.
30:04
Truth was another one that participants mentioned is not as
30:07
easy as you would expect and when you're dealing with use
30:10
cases that have anything to do with compliance or regulation,
30:14
you really have to be careful and it makes it a lot more
30:20
risky. To do anything with large
30:23
language models. Oh and last thing I'll say about
30:26
the ethical concerns is some of the respondents talked about
30:30
intellectual property and how they do not know what data was
30:35
used to train these models and they do not know if they are
30:41
legally able to use the output from these models.
30:46
Considering the data that it may have been trained on could be
30:49
copy written. So a huge challenge that came up
30:55
time and time again in this survey was prompting and how the
31:02
consistency of prompting and what to say, exploring latent,
31:09
space exploring things we find out about a new way to talk to
31:14
these large language models every week and so that is a
31:19
challenge also cash. Caching the prompts or prompt
31:25
tooling we're starting to see more of these companies.
31:29
That is cool to see but prompting feels a bit like it
31:34
could be just an artifact on along the way.
31:38
I'm not sure if the end State really, if you think about us in
31:43
the future 20 years from now, are we really going to be trying
31:47
to still explore late in space through prompts?
31:50
It feels like Probably got a better way to go about that.
31:55
Also when it comes to prompts, is there a nuanced way of
31:59
getting better output? Is there a way of controlling
32:03
the output that is very unclear and that goes back to the
32:07
ethical concerns. Reliable output is something
32:11
that we saw pop up time and time again and reliable output
32:19
through correct. Oops, goes back to those ethical
32:25
mishaps. It's hand-in-hand.
32:29
One of the respondents said that it took us less than three
32:33
minutes to get bio GPT to tell us vaccines cause autism it's
32:40
probably been trained on quite a few Reddit threads that I am
32:43
familiar with or some of my uncle's Facebook data but that's
32:48
a whole nother story. Now the Smaller models are
32:54
easier to control as far as output goes, but they're not
32:58
perfect. So how can we monitor the
33:03
output? How can we make the output
33:05
secure and how can we get that consistency in quality?
33:11
There are certain teams that are looking at output validation
33:16
because they fear the model is hallucinating and the company
33:20
may not be catching it. So so the hallucinations and
33:24
outputs are again. So detrimental to the company's
33:28
reputation, it can cause bad press for the company.
33:32
It can just be embarrassing and depending on the use case, it
33:35
can also be really bad for the user.
33:38
Let's not forget about that. So the big question here when it
33:41
comes to prompting for me is, do we want to try and move to
33:45
something different? What is going to be the end
33:49
state? Are we going to continue with
33:50
prompting or as this? Just some 2023 fun thing that we
33:56
do for a little while. I personally find it fascinating
34:00
that we already have very efficient ways of talking to
34:03
computers through programming, but now, we're just saying.
34:07
Yeah, forget about all that. Let's try and have the computer.
34:11
Understand what we mean when we talk to it and natural language,
34:15
even though natural language can be interpreted many different
34:18
ways. If you recall, The last time you
34:22
spoke to someone or you got into an argument over something over
34:27
text, you can know that the same phrase can be interpreted many
34:33
different ways. So I just wonder is prompting
34:37
the end State, okay, when it comes to some of the technical
34:40
pains and challenges respondents talked about how there was
34:45
rapidly changing apis, and that is just a headache whether it's
34:50
open AI or the newest Shiny tool thing that is moving so fast and
34:55
it's breaking changes are very commonplace.
34:58
There's no such standards and respondents talked about how
35:02
this is not something that they enjoy because then debugging
35:07
becomes a lot harder. And again, with some technical
35:11
pains, the current tooling out there almost forces you to use
35:15
Python which is not as fun for the front end programmers.
35:21
And also it's not as fun. For those who want to go, really
35:24
deep into the lower level languages.
35:28
When it comes to fine-tuning, it's not super straightforward
35:34
and it is not clear how you can do that.
35:37
And also, how much of an increase you get in the model
35:41
output, it's easier to do on smaller models.
35:45
So it may begs the question, are we going to go for Full Circle
35:50
and potentially end? A year or so we're going to all
35:53
be using smaller models again because you don't have to deal
35:57
with these latency requirements or the latency problems and you
36:01
can fine-tune them easier. And the when it comes to
36:04
fine-tuning, one of the respondents said, how can I make
36:08
answers be focused on what my company is doing rather than
36:12
just the general answers. So how do I feed it enough of my
36:16
company's data? So that becomes the default
36:18
answer rather than what it's been trained on before and and
36:22
then on top of that to build on it, how do we keep that
36:26
information of the company evolving?
36:30
As we learn more about the customers and how do we keep
36:34
practical jokers who are conversing with the models to
36:39
not inject some kind of practical joke in there and
36:44
destroy the whole output of the model?
36:46
So the security on that side of things is a little bit of a
36:51
questions. We'll end this goes back to.
36:54
How can we judge the model to give us different output that
36:59
we're looking for? What does it do?
37:01
We need to start creating better keywords or is there a way that
37:04
we can figure it out through late in space and prompting?
37:08
All right, on the business side of things, there were a few
37:12
fascinating questions that came through in the challenges
37:15
section. And the first one is just how
37:18
useful is this really some of the respondents that are not Not
37:22
using large languages in production but they have played
37:26
around with it. They said, look, there's a lot
37:29
of hype out there and we can use this.
37:32
But at this moment in time, it is a headache and to create a
37:38
business case out of it, what do we need to do?
37:41
How can we prove that? It is truly useful and then
37:47
others were saying I just really don't know if I To be using
37:52
smaller models and train it on relevant data or if I should be
37:57
trying to just use the hottest, newest model.
38:02
That is the most performant again on the business.
38:07
Use case. A lot of people highlighted that
38:10
they are not very excited to be dependent on just one company's
38:16
API. They feel that there is a
38:19
monopoly and many respondents It said that they prefer if there
38:24
was some kind of public model with the same performance and
38:28
weights that were public, that would be an incredible relief
38:32
and coming back to this Monopoly idea.
38:35
Some respondents asked, should we pay for a third-party
38:39
provider to train our own model so that we do not have to be
38:45
dependent on an external API? I know that people talked about
38:50
how really, if you Are dependent on, open a eyes, API and open a.
38:56
I for some reason, has an outage boom, there goes your business.
39:00
And speaking of business, the high cost of production is
39:04
something that people could not stop talking about whether it's
39:07
inference cost, its infrastructure cost, its human
39:11
cost of people that actually know how to do stuff with this
39:14
the ROI on using large language models.
39:17
In production is not clear. Now, let's move on to the last
39:21
section, which is open questions and opportunities that the
39:25
respondents had as they went through the big three questions
39:30
that came up from. So many of these answers were
39:34
around fine-tuning embeddings or inference.
39:39
And some big questions that I found fascinating from the
39:43
respondents were, when does the ROI and cost trade off happen?
39:49
For Or open a, i apis versus open source.
39:53
Is it when you're serving 5,000 users versus serving 500,000
39:59
users? When do you think it justifies
40:03
going to open source and doing everything in-house?
40:07
The other big question that was up in the air is just how do you
40:11
decide where to use large language models?
40:14
And who in the organization is helping champion this and
40:18
helping others. Learn how they can augment what
40:21
they're doing Beyond, just the text summarization.
40:26
How do you decide which models to use and what is good enough?
40:31
It's very unclear, all the different models out there and
40:36
also if there's any security vulnerabilities around them, if
40:39
there are any big problems, like the hallucinations you may say,
40:44
oh this model is great at doing x y z, but it has a A big
40:50
knowledge Gap or a big hole when it is talking about this theme
40:56
or this topic. So how do you decide which model
40:59
to use and what is good enough continuing along with some of
41:04
these open questions that it doesn't feel like have been
41:08
solved yet. And that these participants have
41:12
been grappling with is, how are you thinking about tracking user
41:17
feedback in production? How do you get that ground?
41:20
Truth again going back to one of those challenges.
41:24
How do you know that things are working and it's working for the
41:27
better. How are you managing the
41:31
training / deployment of these large language models?
41:36
What are some of the best practices for dealing with large
41:39
language models or foundational models in general?
41:43
Is there anything that you can see as far as trends?
41:48
Go where Where people are saying, look, if you're doing
41:52
this, you might want to have this in mind.
41:56
And then the other best practices around preventing
42:00
leakage for client facing models, that employ online
42:05
learning and to finish up. We have a few stories from the
42:09
participants that I wanted to talk about because I think you
42:14
will get a few laughs out of it. I know I did one of the prompts
42:18
that one of the respondents said, was their new favorite
42:23
prompt. Is that you take, you say,
42:27
whatever it is that you want the large language model to Output,
42:32
you give it the question, your give it the prompt and then you
42:35
finish it off by saying take every input equally, seriously,
42:40
and watch as it fries. Its brain on that.
42:45
And another story from one of the respondents was how
42:50
Microsoft's calls summarization tool can capture a surprising
42:55
degree of detail while hilariously completely missing
43:01
the main point of the meeting. So, it's able to get everything
43:06
that you said, but then when it goes to add action items or talk
43:12
about the main points of the meeting, it it strikes out and
43:18
so there you have it. This was a little bit of a recap
43:22
of the report that I'm trying to write right now.
43:24
I just thought it would be nice to have it in podcast form if
43:28
you are like me and you love podcasts, hopefully this has
43:31
been quite useful to know about who is using large language
43:34
models in production. What kind of use cases they are
43:37
using them with if there's any special toolings what kind of
43:41
questions are around that the challenges and concerns the
43:44
major? Challenges and concerns, and the
43:48
open questions, and opportunities that still can be
43:51
fleshed out. Again, going back to those eight
43:54
main points. Just so that you have them in
43:58
the top of your mind. Common use cases, like text
44:01
generation and summarization, or useful, but participants are
44:05
going deeper. And they are exploring ways to
44:07
use large language models for data enrichment with data,
44:11
labeling augmentation, synthetic data creation, and Question
44:16
generation my personal favorite number two decisions to use
44:20
large language models within an organization is still unclear
44:23
due to the high cost and unknown Roi.
44:27
Remember the high costs are anything from serving cost to
44:32
infrastructure cost to human costs resources.
44:37
All of that it's just an unknown Roi right now and it is also not
44:43
so clear how to prove out. Roi number three, you have the
44:47
challenges around using large language models.
44:49
Speed of inference, makes certain use cases, absolutely
44:54
impossible. They are off the table for
44:57
current infrastructure is not designed for these massive
45:01
models. And there are a lot of
45:03
workarounds and quick fixes one of my favorite responses to this
45:08
was hey are you using any special tools?
45:12
And someone put nope, just a bunch of shitty Scripts That I
45:15
am not looking forward to refactoring.
45:18
Number five is companies are building custom tooling to help
45:22
with pains around output validation and prompt tracking
45:27
and going along with this prompt tracking.
45:29
For number 6, hallucinations present Real ethical and
45:34
business concerns so much so that it's a showstopper for
45:39
certain use cases and certain companies.
45:42
Number seven is that there are larger question.
45:45
Such as which part of the current workflows will be
45:48
replaced versus, which parts will be augmented.
45:52
And there is no Clarity around that whatsoever.
45:56
And the last one is prompting and how, valuable of a skillet
46:01
is. But even the best prompts cannot
46:06
force. The models to give reliable
46:08
output there. You have it.
46:10
Hope you enjoyed this, very special edition of our large
46:14
language. Dolls in production podcast.
46:18
Join us on Thursday April 13. We've got over 33 speakers that
46:25
are going to be there. And we're also doing two
46:29
workshops all around using large language models in production.
46:34
I think you're going to like it. If you liked this podcast if you
46:36
made it this far come see us say hi.
46:40
I'm going to be also doing a bunch of shenanigans and maybe
46:45
Maybe playing a few songs that I can prompt the crowd to Give me
46:52
lyrics for not chat GPT. All right, that's it.
46:56
Have a great day, everyone.