Latest / Agentic Conversations (formally mlops.community) / Impact of LLMs on the Tech Stack and Product Development // Anand Das // #188

Transcript

Impact of LLMs on the Tech Stack and Product Development // Anand Das // #188 · Agentic Conversations (formally mlops.community)

0:00
Amandas and CTO and co-founder of Bitto and I like my coffee
0:06
Black Extra strong triple shots of espresso which a little
0:11
amount of water in the morning. All right, all right everyone.
0:17
We are back for another ML OPS community podcast and what a
0:20
podcast we have today. If you are into MLM's and
0:25
figuring out how the hell you can get them to work properly,
0:29
this is just the podcast for you.
0:31
Anandas was so forthcoming on how he spoke about what they're
0:37
doing at Beat to their AI coding assistant and how he has gone up
0:43
against some severe challenges. And they are at this funny place
0:48
where they don't want to go open source and they don't want to
0:51
start creating their own GPU racks because that is quite
0:55
costly. But they are hitting every limit
0:59
that you can think of when it comes to their API calls because
1:02
of the size and the scale that they're at.
1:05
They are doing things like using four or five different APIs.
1:11
So they're they're using ChatGPT or 3.5 through the Open AI API.
1:16
And they're also going through Azure and then they're using
1:19
Open AI or Chat. I don't even know how to say it
1:23
these days, but GT4 and then they're using both anthropic
1:28
models and. He was super transparent on how
1:34
they look at. Which model they're going to hit
1:38
and also the load balancer that they've created so that they
1:42
don't get these rate limits that some of us are very accustomed
1:46
to and it's the good old rate limits that will bite you in the
1:49
butt. He also talked about how
1:52
important it is to have guardrails and I thought it was.
1:55
Cool to see that he really likes to be super concise with the
2:01
prompts. An in his mind the guardrails
2:04
needed to be really, really heavy on the prompting.
2:09
So on the guardrail side, it was fascinating to see how he put a
2:14
lot of emphasis in making sure that the prompts are right and
2:18
you're giving as much context as you can.
2:21
And the prompting side, try to dig in and see if he's got a
2:24
vector database if they're using any prompting.
2:26
Rules what the actual stack looks like, but for now they've
2:30
done it all homegrown and it is open source.
2:32
O an undas, the CTO of B2 was incredible and transparent and
2:38
everything that you would want in an interviewee because he
2:43
just opened up the kimono, as they say.
2:46
Let's get into the whole conversation and before we do,
2:49
you know what it is. It means the world to me if you
2:53
can just share this episode with one person that you think would
2:57
find it. Valuable because sharing is
3:00
caring. All right everybody, here we go.
3:09
An undas, it is great to have you here.
3:12
I'm excited because you've got quite the track record.
3:15
I mean, you were with a company that went public in the 2000s
3:20
and then you were the CTO of a company that got bought
3:24
recently, more recently I guess, than the early 2000s, which was
3:29
in 2021. And now your new endeavour
3:31
Beats, so is kicking off strong. And so I want to hear about the
3:37
journey. Like, how did you even get into
3:39
tech? How did you get to where you're
3:41
at right now? So I started straight out of
3:44
college. I did my Masters in Computer
3:46
Science from University of Pune and got the first internship in
3:50
Veritas, which was, you know, completely different rule.
3:53
You know, we're building file systems and volume management,
3:56
right? How to store bits, bytes, files
3:59
on discs? It was very difficult to explain
4:03
what I'm working on to my friends and my family at that
4:06
point in time. But you know, like we knew
4:10
everybody used us, but, you know, I never got a chance to
4:13
basically build something that uses, you know, kind of use
4:17
right front and centre and they know what it means and how it
4:21
works and so on, right. So I wanted to kind of try
4:25
something different. And I was like since my college
4:28
days I was interested in compiler construction.
4:30
So I'm going to have left Veritas in 2005, worked on a
4:34
bunch of products out there and joined the Panther sisters,
4:38
which is into high performance computing.
4:40
They're building great computers and they need it like compilers
4:43
which can take normal code and then, you know, paralyse it.
4:47
But as the story goes, they also started working on like storage
4:50
related problems. I was back into storage.
4:53
I really wanted to do something different and that's when I met
4:58
Amateur and Rajiv along with my, you know 10 friend Mukul, we
5:04
started Pubmatic. We just completely different
5:07
concept like it was ad tech and we are building a platform to
5:11
generate more revenue for publishers.
5:13
The first Sunside platform, you know 14 years, 15 years later,
5:18
you know it kind of went public. You know, I'd say it was a team
5:21
effort, learned a lot about scale, data processing, you name
5:28
it like we are running on low margins and you have a lot of
5:32
data and transactions to process.
5:36
And you know, after the team effort, we went public.
5:39
I joined another company known as IOTA, primarily for two
5:43
reasons, right? Once you're in an organisation
5:45
that you've founded for more than 1214 years, the problem is,
5:52
over a period of time, although you think you're working, you
5:54
know, it sometimes feels like you're on the couch, everybody
5:57
listens to you. So I wanted to kind of try out
6:00
something different and also work with people that I don't
6:02
know and see I still have it in me to, you know kind of change
6:07
the organisation or drive change in the organisation, make it
6:09
more performant. And you know it worked out well.
6:13
I would have got bought by DNB but learned a lot on development
6:18
at that point in time where we spent time.
6:20
There are different types of people how to kind of like push
6:24
them through and so on. But you know, learned a lot
6:27
about technical challenges that everybody faces, right?
6:30
Like once you know the whole platform, it's easy.
6:32
But when you're going in straight into a different
6:35
organisation, you have to understand the platform.
6:38
You have to see who's the right choice for the right issues and
6:44
modules that you need to build. And Amir and I were thinking
6:48
over a period of time like to build something.
6:51
That is where we started Bitto. And the interesting thing about
6:54
INTO is we have already pivoted, right?
6:56
A lot of people don't know, they kind of CSS and AI too.
6:59
But we started with a very simple concept of, you know, how
7:03
do we make developers collaborate, right?
7:05
Because we've been running organisations for in dev teams
7:08
are not in one location, right? So even though there's no work
7:12
from home at that point in time, your developers in the US, your
7:14
developers in Europe and your developers in Asia, and they are
7:19
all working through the office, but you know the communication
7:23
is lacking, right? Like, you know, yeah I build
7:25
this, but you know they went home and like the other guy
7:27
doesn't know if it's built or not, they're solving the same
7:30
problems or you know this code review which is pending for two
7:33
days just because of like depends on issues.
7:37
So we started with like, you know, how do you get Slag and
7:41
Google doc combination in the IDE, right?
7:44
We got like, really good feedback.
7:47
And then figure out nobody's using it.
7:49
So, so then a classic case, right?
7:54
Like you go talk to people, you figure out what they want and
7:57
then you figure out that, you know, they have a lot of
8:00
requirements, but when it comes to using it, it's really not
8:03
there where they can use it. And we figured out that most of
8:07
the people are used to muscle memory of using Slack.
8:10
They'll copy paste code out there and and stuff.
8:13
Although it's difficult to find and so on.
8:15
But you know, people get used to it.
8:18
But we started hearing a lot of roundup requirements about how
8:21
do I onboard new engineers, right.
8:23
It's very difficult to explain code.
8:25
Senior engineers don't want to spend time bringing up.
8:30
It's not that they don't want to spend time, they don't have time
8:33
because they have to deliver on the product Rd maps and it just
8:37
sucks energy and they have to go and explain someone that they
8:40
kind of can talk about in their sleep, right in detail.
8:44
And some people were like, can you have to generate documents,
8:47
If somebody has a question, can you automatically answer it?
8:50
And that is what led us to kind of look at generative AI, right
8:54
At that point in time, I would say GPD 2 was out.
8:57
It was not really quite there. And then suddenly cheaply 3.5
9:01
came out, which was awesome because it had reasoning
9:05
capabilities. It could basically answer on
9:08
general topics, although you know, we figured out very early
9:12
on that it helps and it's quite a lot, right?
9:15
So it will cook up things. So we had to put some
9:18
guardrails, but that is where we kind of pivoted inside.
9:21
Hey, what if we built a assistant that anybody in an
9:27
organisation can use or a normal developer right can use to start
9:34
getting explanation about the code that exists right?
9:38
Yeah, and we started with a very simple thing to test out.
9:40
What does Replica simple Chrome extension with a single menu
9:44
option explain code. That's it, right?
9:48
And like, you know, rather than asking people that we knew
9:53
whether you want to use this or not, we just launched it.
9:55
Like, you know, let's see how many people download and use it
9:58
and do a surprise, a lot of people actually downloaded it
10:01
and started using it. We stored as that, OK, there is
10:03
a need for this, right? And then we immediately launched
10:08
our extension on Visual Studio Code as well as checkpoints as a
10:13
personal assistant, which actually helps you generate
10:16
code, explain code, write test cases and so on.
10:20
And that's what led us into this area and we've been physically
10:24
expand. So you started out and you had
10:28
this deep desire for optimization and you went into
10:31
the compilers. You then were trying to optimise
10:35
companies and now it feels like you're optimising developers in
10:39
a way. And so there's I see a theme
10:42
there and that's coming through. It's quite nice.
10:45
How much of this whole optimization piece do you feel
10:49
like you are still trying to tackle?
10:52
And are you doing things on these lower levels right now
10:55
where it's like? I know that before we hit
10:58
record, you said Ohh, yeah, I have some.
11:01
You have some GPUs lying around the office and are you
11:04
tinkering, trying to analyse and optimise at those levels where
11:09
you at right now? Like what's what do you get into
11:12
for fun in this optimization level?
11:15
So the fun aspect is trying to figure out like you know Open AI
11:21
and Tropic and all these guys are running huge models like
11:25
tonnes of GPUs. And you know, sometimes when
11:29
you're answering questions related to a particular topic or
11:33
you know say for example your own code, do you need that
11:36
become model right. So can I actually have an open
11:41
source model which is rather than seeing training, providing
11:45
context on what I have right now and can it answer questions
11:50
relatively well on lower cost, right.
11:54
And the primary reason for doing this is not like, you know, just
11:58
basically saving on cost, but you know, in the future, in the
12:03
future or even right now for enterprises, you know,
12:05
everybody's kind of worried about tech going to L models,
12:09
right? And you know, it depends upon
12:12
whether you remember to fill the form, don't use my data for
12:15
training and stuff. If somebody forgets it, then
12:17
your data is going out. Obviously it's not going to be
12:19
publicly published, but nobody wants their hard earned IP to be
12:24
there with someone, right. So most of the people are kind
12:27
of like enterprises and some people who are worried about
12:30
their code are more interested in, and I actually have
12:33
something that I can host locally in my data centre,
12:37
right? And we are not really there
12:39
right now. You know, there are models which
12:41
you can run, you know, people say on your laptop and stuff.
12:44
But The thing is if you run it and you try to ask questions and
12:47
get answers, it's not really there.
12:49
It's not basically writing your answers in the time that you
12:52
would be happy with, right? It will return your answers but
12:54
you'll go grab a cup of coffee and come back.
12:57
That's not how it works. So but I'm trying to figure out,
13:01
you know, what are the tricks out there?
13:05
Can I optimise? Like fine tuning, right?
13:08
We are not kind of using fine tuning right now, but you know,
13:12
our idea is at some point in time we'll need to fine tune
13:14
some models. So you know, learning about it,
13:17
testing waters. So that is, that is what
13:19
tinkering I'm doing right now I'd say.
13:22
And it sounds like you're using a whole suite of models.
13:25
And whether or not so are you basically saying you're going to
13:31
try all the different APIs as a service, like the open AIS and
13:34
the Anthropics and coheres whatever it may be and you're
13:38
trying to also leverage the open source stuff that's out there?
13:42
Yes, yes. So we are leveraging open
13:45
source, I would say for embeddings right Now, you know
13:47
the models out there that we would like to use to answer
13:51
questions. You know, we need to host a lot
13:53
of GPUs, right, for a startup at our current stage, you know,
13:58
it's not feasible, right? So it would be using, I'd say,
14:03
commodity apps as I call them right now.
14:06
So you have Open AI, Entropic. You also can get Open Air
14:10
through Azure, right? And the other a VPC.
14:14
Amazon has come off with Petroc and Amazon has its own models.
14:18
So does Google and early on when we started building the
14:24
extension and you know we saw a steep increase in usage and we
14:29
had like one Open AI account at that point in time and we
14:33
started hitting like TPM and RPM limits left and right.
14:37
Yeah, you know and obviously there are small team also,
14:40
right. Open AI is expanding like
14:43
they're a good technical team, small team and really focus.
14:47
That said, you know if you if you make a request to basically
14:50
increase your TPMS and RPMS, you know if it's small enough,
14:55
they'll do it. But most of the times, like you
14:58
know, at some point in time they come back and say that's the Max
15:01
I can give you. So you know you need to have,
15:05
you know, fall back options, right?
15:08
So what if you hit 3.5 limits? What if you hit GPT 4 limits?
15:12
What if you hit, you know, anthropic limits, right?
15:16
So we actually like, you know, build a simple mustard mugs and
15:22
load balancer, right? So what we do, and this is not
15:28
like anybody who goes into this problem will actually build
15:30
something like this, right? So we have multiple accounts,
15:35
right? And you basically have different
15:38
CPM and RPM limits on them. You need to decide anytime the
15:42
request comes in, do you have enough TPM and RPM to handle
15:46
this request and which account I should use, right.
15:48
So if you're using just one other then you can utilise that
15:52
mechanics. Obviously you can't have in a
15:55
huge number of accounts with one, right.
15:57
It has to be relatively enough so that those guys are OK, you
16:01
are OK. And then you have to manage
16:03
costs across all of those because you have your credit
16:06
cards being charged. And then if you have issues like
16:11
models do go down, right? And people are OK with that
16:14
because there's a new technology and things can go wrong.
16:17
So nobody's like upset, but the same point in time, somebody
16:21
who's trying to get an answer right now, if he doesn't care it
16:24
he's pissed. So you need to, you know, kind
16:27
of make sure that OK, if one model is not working, which is
16:31
the next best model that I can use, right?
16:34
Yeah, so we need like options to kind of failover within multiple
16:39
accounts like you know say opening a direct, opening it
16:42
with Azure, right? Can I feel over between those
16:45
two and this two at fun time? And then if these two are not
16:50
available, can I use Entropic? And then Entropic has multiple
16:54
models. You have like cloudy, instant if
16:56
cloudy. Guidance is pretty fast, but you
17:00
know it's fast and furious sometimes might like it doesn't
17:04
give you the right answer and if you use Kodi two, you know it's
17:08
great at the same point in time it takes slightly longer time to
17:11
understand things and there. Are much bigger answers, Yeah.
17:15
And the costs are different, right.
17:17
So you have to kind of manage those aspects.
17:20
So when you kind of look at all of this, one is like, how do you
17:22
make it highly available? How do you make sure that when
17:26
the request comes in in a, you're basically using external
17:29
APIs which can be up and down South, identify which one is
17:32
live and whom should I send it to based on the request that is
17:36
coming based on the size of the request.
17:39
Because if it can fit in into 4K I can give 3.5 a request or if
17:44
it basically goes to 16 K now I have 3.5 its support 16K but if
17:49
it is more than 16 Ki need to use either GD4 or I need to use
17:52
cloudy O. You know you need to do that
17:54
switch and at some point in time you need to also degrade the
17:58
functionality. For example, there is nothing
18:00
else available beyond like GPT 3.5.
18:03
If the actual question was less than 4K then you say OK, I'll
18:09
reduce the context that I give it, but I'll try to answer the
18:11
question as much as possible. So a lot of things actually go
18:15
in, like when people say using LLM is simple, we need kind of
18:20
creating a solution that is generally available.
18:24
You know, there are a lot of hoops that you have to jump
18:28
across to actually make that happen.
18:30
OK. And it feels so janky that you
18:33
have to do it like that. It's like such a hacked way to
18:37
make it work. And it I guess it's just it's a
18:41
testament for how. Popular these models are because
18:46
with all the money that's being thrown at Open AI and Anthropic,
18:50
they still can't service the traffic that they have and the
18:54
demand that there is, which is incredible to me.
18:57
But there's also this idea of. The evaluation of the models,
19:01
and you talked a bit about that, you kind of touched on it.
19:04
It feels like you're looking at a few things.
19:08
I am guessing you have like the gold standard model that you're
19:12
always going to try and hit 3.5 if you can because that's
19:16
fastest, cheapest and most accurate or best out there.
19:20
Or at least there was. But I don't know, like every
19:23
once in a while they update it and it's a little bit like I
19:28
think an open source model might be better than this.
19:30
Right now and then they have to re update it or whatever, but
19:33
that's a whole nother story. And so when you're looking at
19:37
the evaluation you have, OK, does it fit into this 4K
19:39
context? Then I'm going to try and hit
19:41
3.5. If it doesn't, I'm going to try
19:44
and maybe hit GPG 4, but I have to be.
19:48
Wary of a few things, A it's going to be much slower and then
19:52
B it's going to be much more costly.
19:56
And the same goes with anthropic.
19:59
If you start throwing it over there, do you have a?
20:05
Rough idea of like the decision tree on which model needs to hit
20:14
and how that decision is being made.
20:18
So primarily you know right now we do kind of manage cost,
20:23
right. But you know that is like you
20:25
know I'd say the last priority to kind of look at because
20:28
you're basically trying to make sure that the user gets what
20:32
they want. So primarily it is based on
20:34
couple of things, right size of the input, can I process the
20:38
input with the algorithm and then which algorithm is good for
20:43
it. So for example if you're asking
20:47
normal question. And you have a big chat history.
20:51
I'd like to give as much context as possible, because sometimes
20:54
when you're kind of generating code, you know you ask a
20:56
question, you get the code you want to make modifications, so
20:59
you're asking questions based on the context, right?
21:01
So I would like to keep that context as much as possible.
21:06
So out there, based on the context size and your prompt
21:09
size, I would pick a model, right?
21:13
That's the first thing. But the second thing, which is
21:16
again important is is the model OK?
21:20
For answering that question. So for example, if I'm using
21:23
GD4, I'll get better answers, so it's better at reasoning, right?
21:28
If I use GPT 3.5 it's good, but at the same point in time there
21:32
are certain things that it will hallucinate, move on or it won't
21:35
give you proper out. A very simple example would be
21:38
give me a shell command to do X and it will give you shell
21:42
command and also print a lot around a blurb about how to use
21:45
it and so on. But sometimes if people want to
21:47
actually do something like, I would like to ask you a shell
21:50
command to list something and I just want to get the output of
21:54
the shell command. I want to parse it out and run
21:56
it. Just not possible, right?
21:59
But if you basically use GPU four and you give proper
22:01
instructions in the prompt, you can do that right?
22:04
So depending upon the instructions right and the use
22:07
keys, you need to basically decide whether this particular
22:09
model is OK, or even if some model is costly, you need to
22:13
move to that. Right.
22:15
Yeah. Now, you know, we recently
22:18
supported, you know, a I that understands your code, which is
22:21
we index your code, we generate context, right?
22:25
And LMS are very good at answering general questions.
22:28
But then you can basically put some garden seeing that this is
22:31
it, this is the context, using this context answer this
22:34
question, right. So I need to generate that
22:36
context. That context can be huge.
22:38
I can't use GPT 4. Right activist Keiko to Cloudy
22:43
2:00 so that I can give as much context as possible.
22:46
And what we figured out is even that is not enough sometimes
22:49
right to answer the question concretely.
22:51
So. Number one is size #2 is the
22:55
kind of operation that you're doing and is the model good
22:58
enough to actually handle that, right.
23:00
These two become primary, I'd say, constraints based on which
23:04
you decide the model and obviously.
23:06
You have like a map to degrade the functionality.
23:10
Like if I don't have TPMS on GPD 4, right?
23:13
Yeah, OK, the next best is 3.5 I don't have out there.
23:16
Then I'll be simply switched to cloudy, right?
23:20
And that degradation is based on the the lowest denominator,
23:25
which is like the cost right from the user's perspective.
23:28
So those are the ways we kind of deal with it.
23:31
Well, and now walk me through the other pieces in the stack
23:34
that you have, because I imagine at some point you're doing some
23:41
retrieval augmented generations too.
23:43
So is there a vector database in there?
23:45
And you mentioned guardrails. What kind of are you using tools
23:51
for these guardrails? How are you making sure that
23:53
there are no hallucinations? And what other tools are you are
23:59
you using to help? You just have the best answers
24:02
that you can and really the more confidence that you can have
24:08
when you are letting these users say, tell me what this means or
24:13
give me this shell script like you were saying earlier.
24:17
So you know it's the first question about like you know
24:20
what is the stack that we're using for you know, a I that
24:24
understands your code. So do we use vector delivers.
24:27
The answer is currently we use like our own homegrown.
24:32
I will call it a vector database, but you know a map in
24:36
memory we'd be soon moving to a database.
24:39
The reason why we actually did that is the current
24:43
implementation that we have. We want to make sure that.
24:47
Users data stays on that machine, right?
24:50
So we don't index and move anything to our server side.
24:54
We don't touch your code and we don't move anything to our
24:57
server side. So we wanted something which
24:59
actually installs as an ID extension, like indexes stuff
25:04
directly on your machine, so the index file is stored on your
25:07
machine. And in memory I need the same
25:12
mechanics of something like say Pinkun for example, right
25:15
insert, upsert and so on. So we built up one small layer.
25:20
On top of it that we utilise which is in memory, right?
25:25
Then it again has its own restriction right?
25:27
Because all these guys, if you're running it within the IDE
25:32
depends upon whether you're running it within a web view.
25:36
If it's within a web view, you have a limitation of 512 like a
25:40
lot of restrictions. So you have to kind of make sure
25:42
that first of all I'll support like very basic case like I
25:46
don't believe that repositories will be.
25:48
I don't support repositories which are.
25:50
To it 120 MB, right, Let's start with that.
25:53
Now if you take that repository, if you're going to index and we
25:58
are not a cool search company to start with, right.
26:01
So we are going to add symbol search as we kind of go ahead.
26:05
But right now we have like embedding based indexes and
26:09
those embeddings are vectors. Those numeric vectors, when you
26:15
load it in memory is smaller in size.
26:17
But when you store it on a file system, yeah, it kind of
26:20
increases the size, you know, sometimes by 3X, right?
26:22
So your index might be more than you're cool.
26:24
So we use in memory because we don't want people to install
26:28
something more. But over a period of time, what
26:30
we are kind of looking at is can I basically have a vector
26:32
database which is easy to install, available as a library,
26:37
And if there is none then we'll have to build one depending upon
26:42
how soon we hit. Just got flights but on the
26:46
server side idly will use one which will basically give tester
26:50
pine cone a bunch of others but you'll use one which is like can
26:54
we hosted locally can scale up and scale down.
26:59
We do use, but it's like home and I would say just for the use
27:02
cases that we need. Makes total sense.
27:05
Yeah. And that's a whole.
27:07
Another can of worms that I bet you have to work through is how
27:12
to ensure privacy when you've got this and make sure that
27:16
people are OK with you knowing all of their code and especially
27:22
for these non open source projects.
27:24
And if they're going to ask you questions and they have to be
27:29
willing to, I guess give up a bit of privacy because maybe
27:35
you're not doing anything with a vector database, but you are
27:38
sending the code to one of these companies, these API companies.
27:44
And so I guess that's where the beauty of getting local or open
27:51
source models comes into play later on down the line.
27:54
But what about I I don't want to skip over the idea of what about
27:58
the tools for the guardrails, are you using anything there?
28:02
And how are you assuring that the output is actually valid or
28:06
not hallucinated? I'd say that we are not 100%
28:09
there, right. It's still 100 needs.
28:11
At the same point in time, the gatherers that we have in place,
28:15
we are not using any kind of tool right now, but tools that
28:21
you use will be placed on certain use cases.
28:23
I'll go into one. But right now what we're doing
28:27
is we're kind of looking at the answers that we are getting,
28:30
right. Sometimes, you know, people
28:32
basically put a thumbs up and thumbs down, right.
28:34
Which also means that they would like to share some data with
28:37
better saying that, hey, this is the case, you know, just didn't
28:40
really work wherein if we basically have the context of
28:44
what was there from the index and what answer was given or
28:49
sometimes, you know, there is a hallucination, actually figure
28:52
out why is there a hallucination, right.
28:55
And most of the times, as you know.
28:57
Alms are very good, but similar to any computer programme.
29:00
Garbage in, garbage out. So if you don't ask the right
29:02
question. You don't get the right answer.
29:05
Right. And if you have a natural
29:08
language interface that people will ask anything because they
29:11
think they're asking the right thing, but they haven't given
29:13
enough context or like you know, enough Nuggets to LLM to
29:18
basically figure out what is it that you want to get answers to.
29:21
So when we get something like that, then we kind of look at do
29:25
I need to change the prompt that I have or I need to put some
29:28
extra rules for it to actually answer And then we update the
29:32
rules accordingly and then periodically we.
29:36
Keep on running tests on the model.
29:39
So that we keep on like updating that checklist, right?
29:43
That is what we're doing right now.
29:45
But I'll take another use case for example, right?
29:48
So say that you want to do security analysis of your code,
29:51
right? And you want to basically check
29:53
for open source vulnerabilities. Now if you only use Ellen, it's
29:58
going to give you some data. It's not going to say I'm not
30:00
going to tell you what security issues are there.
30:02
It will give you some security issues.
30:04
Some security issues are based on the data that it has been
30:06
trained on and some it will help me.
30:09
But out there, the right approach would be to actually
30:12
take and run a open source vulnerability tool.
30:16
Right which is very easy right? Like you know don't use rocket
30:18
science. This is available today.
30:20
Just scan the code you get the output from open source
30:23
vulnerability tools. Once you get that then feed it
30:27
as a context to LLM and say these are the vulnerabilities.
30:31
Tell me where this occurs and or like it also has the information
30:35
of where these are not. Tell me how to fix it.
30:38
Right. That is a better implementation
30:41
wherein you know you're already putting guardrails in place,
30:44
right? Using tools which are
30:47
deterministic, right? Not basically, just depending
30:52
upon like generative AI in that scenario, right.
30:55
So you know that is how you put some guardrails and I would say
30:59
like hallucination is there, but sometimes I believe
31:01
hallucination is a necessary evil.
31:04
Because if it just answers based on whatever it knows and just
31:09
repeats what it knows. Then it's not generating
31:13
something new. Right.
31:15
So you know, it's like you're telling somebody, like, you
31:18
know, read this book and Justice answer from within this book,
31:20
he's not going to think beyond that.
31:22
We want it to think beyond that, right.
31:24
So it's kind of a necessary evil, but people are trying to
31:27
actually utilise it to get context analysis on something
31:31
that is existing. And the only problem is like
31:34
nobody is like, you know, it's not trained enough.
31:36
Like it's trained enough, but it's not trained enough to
31:38
basically go and say if I'm talking about something which is
31:41
existing, let me go and verify. Right, using existential
31:46
information, what I'm saying is right or wrong, right.
31:50
And it will come in the future because you know, like over a
31:53
period of time like we have solved, we have nowhere there.
31:56
Now we have something which can actually answer.
31:59
So now it's like a kid, you have to train and put some guardrails
32:02
so that it starts behaving properly over a period of time.
32:06
Well, I like this idea that you're really on the prompt
32:09
side. You're trying to figure out
32:12
everything you can do before feeding the question or the
32:15
context to the LLM. And so you're trying to say,
32:19
hey, can we solve this with heuristics?
32:22
Can we solve this some other way and then give that to the LLM?
32:26
Or maybe you just don't even need to end U giving it to the
32:29
LLM and you can just get your answer and you're good.
32:33
And so yeah, it's it's nice that it's looking at it on the prompt
32:37
side as opposed to looking at the output and seeing what the
32:40
output is. And I guess you're getting
32:43
flagged when you get those hallucinations with the thumbs
32:46
up, thumbs down. And it feels like that you've
32:48
got a human in the loop that goes and is manually trying to
32:52
evaluate what happened. Yeah, Which is another one.
32:56
Yeah. I think that's pretty on par for
32:58
what we've been hearing. And I bring this up because we
33:01
have a whole survey out right now that's all around LM
33:06
evaluation and how people are doing it and what they're doing.
33:09
And so I think the thumbs up, thumbs down is a huge one.
33:12
And then getting a human in there and figuring out where
33:15
things went wrong is another one.
33:17
That is, these are very clear indicators of trends that people
33:21
are doing right now. And so it's it's what we're
33:25
using. I don't know if it's perfect.
33:28
I don't think it's perfect by any means.
33:30
I should clarify that. But it's like you said, it's
33:33
nice that it's a necessary evil sometimes and you got to work
33:36
with it. Yeah.
33:38
And like, you know, sometimes you know why I say it is a
33:41
necessary evil is because you know, think about this, right?
33:43
You don't have. An EPI.
33:46
But you know, you're kind of like just giving your concepts
33:48
to the LLM to get an answer like, hey, I want to build this
33:52
and I want to have an API and it will generate an API for you,
33:57
right? And sometimes people are like,
33:58
keep me an answer, this EPI doesn't exist.
34:00
But if you look at it the other way around, somebody who's
34:03
actually trying to build something gets an EPI design
34:07
just because somebody is needed. Now it's like, OK, now I have an
34:10
API design, let me implement it this way.
34:12
So there are two sides of the coin, right?
34:13
Depends upon like if you are. Basically they gonna somebody's
34:16
like I want to make changes to an existing code out there.
34:20
That answer is wrong. But if somebody's kind of
34:24
generating new code and wants that idea, that answer is right.
34:27
But the LLM basically is going to answer based on what it knows
34:31
and the context that you've given and the gardens.
34:33
So if those things are not proper right, the same answer
34:38
can be wrong for one, right for another.
34:41
So that's the interesting aspect here.
34:43
Not to get too philosophical, but that is awesome to think
34:48
about. The other piece that I was
34:50
wondering about because prompts are so important for you and you
34:53
mentioned testing prompts. How are you tracking prompts?
34:56
How are you testing these prompts?
34:58
Are you using a tool on that side?
35:01
So right now we don't use any tool on that side.
35:04
We are kind of checking out a lot of open source tools that
35:07
are out there, right, which can basically run prompts on
35:12
different inputs, look at the outputs and then basically check
35:16
them or verify them. Our main issue is with the
35:20
verification piece. How do you verify something that
35:24
you know? You don't know what the answer
35:26
is because like even human in the loop, right?
35:29
Say that I'm basically looking at a technical question.
35:33
You know, there are few people who are familiar with like, you
35:35
know, crushed us basically picking up.
35:36
But you know, that's like, you know, there are a lot of Java
35:39
guys who are not familiar with REST.
35:40
Now if I'm having him verify it. He he, like you know, human in
35:44
the loop is also incapable sometimes.
35:47
And then when you depend upon the LLM, The thing is it can
35:50
hallucinate when you're basically matching the results,
35:54
right. So that's why what we have is we
35:59
have like, you know, based on all the thumbs up, thumbs down,
36:01
whatever we get, plus some test scenarios that we create in the
36:05
data that we generate. You basically try to figure out
36:09
what is the right possible set of answers and then basically.
36:14
Response on those so that you know the test results are.
36:19
Recrafted, Verified. Yeah.
36:22
So that we don't depend upon like AI generating an answer and
36:26
EI testing it with what it thinks is right.
36:30
We've kind of figured that that can pause.
36:33
His seat is issue. That in, you know, like the
36:37
normal human thing, right? Or like, you know, you want
36:40
something to be right. So you think about it and then,
36:42
you know, you basically come up with all the reasons why that
36:44
is, right. We don't want to have the same
36:46
thing happen with like our testing process.
36:50
So we run it. So it's a homegrown tool.
36:53
But at the same point in time, I not say that it's just perfect
36:56
by any means. And we also get inspired by the
37:00
open source tools which are built out there.
37:02
You know, some people can put some novel ideas and then we are
37:05
like, OK, how do we bring this? Except two or two?
37:08
So that is how we're kind of doing it currently.
37:10
But I believe that some of these tools which are kind of working
37:14
in that direction over a period of time will solve for those
37:17
problems. So we don't have to kind of deal
37:19
with it the progressing pretty well.
37:22
But that said, you know, there's like there's this gap that we
37:26
think needs to basically be filled before.
37:29
It can be used like. As something that he can use in
37:34
production on a daily basis. Yeah, yeah.
37:37
We had the creator of Airflow who also created Apache preset
37:44
and also created Romp Demise recently on the podcast and
37:49
that's one of them. That's why I was asking because
37:51
he was very much talking about how, yes, we need to test our
37:55
Proms and we need to look at it like you have test driven
37:59
development and now this is prompt driven development and I
38:04
I like that idea. I do know that there are a lot
38:07
of companies that are trying to tackle this and each one is
38:10
going at it in a different way and there's still a lot left to
38:14
be desired because again, everyone is doing it in their
38:19
own flavour and so it's not quite like you can have one tool
38:22
that solves problems for everyone.
38:24
Yeah. And I think, you know, say one
38:27
year, two years down the line, right, because these companies
38:30
are like, you know, like everybody like say air flow guys
38:33
and other guys are gonna build homegrown solutions, there'll be
38:37
open source solutions and there might be 10 problems that we
38:41
need to solve somebody solving 5.
38:42
And these guys are, you know, kind of focused on like there
38:44
are one or two problems like us. Yeah.
38:47
Then you basically look a year down the line, all these
38:50
problems now become common problems and then you have
38:53
solutions at different places. Then you can combine and create
38:57
one or at least that knowledge is there.
38:59
So nobody has to reinvent the wheel.
39:00
But you kind of put all of these things together, so it will just
39:04
going to get better from here on I would say.
39:07
Yeah, completely, completely. What are some other prompting
39:10
best practises that you've found?
39:12
I imagine you've been doing a lot of it over the last couple
39:15
of months, trying to figure out a Are you prompting 3.5 and how
39:20
that differs then when you prompt four, or when you're
39:23
prompting anthropic, or when you're prompting a Lama model,
39:27
like, how do you deal with all of that complexity?
39:31
So you know, we start with something like a base model like
39:35
you know, prompt, I would say running it on 3.5 and you know
39:38
you're kind of tuning it to get a certain result.
39:42
Then you kind of run it on, you know, GPD 4.
39:45
Most of the times it will run perfectly fine on GPD 4, but
39:48
sometimes you have to tweak it. 3/4 is very good at structured
39:54
prompting. Right, what we kind of figured
39:56
out like if you give the GSON of YAML, kind of like, you know, a
40:03
structured format structured like rules, then it's very good
40:08
at evaluating those. Now if I put the same thing in
40:10
3.5. You know, there are certain
40:13
things that it won't do properly.
40:15
For example, they get us said. It's going to give you some
40:18
blurs that you don't want, and you have to do a bunch of things
40:21
to actually remove that, right? Sometimes you just see that
40:25
that's enough ranting. I'll basically put like hardcore
40:28
logic to just pick U things, You know, I'll put markers, I'll
40:32
give a prompt so that put markers so that I know I have to
40:35
just pick these things using the markers and remove the other
40:38
things, right? You have to do that at some
40:41
point in time people are like that's enough prompting and in
40:45
case of Entropic also is the same thing.
40:49
Like you know if you use a GPD 4 prompt directly in anthropic it
40:53
won't give you the same results. So you have to kind of change
40:55
the product. So for any given option, to be
40:58
very frank, we have for every LLM is separate prompt, right?
41:03
Like you can share the prompt if it works, but most other times
41:06
you have tweets which are very specific to that LLN so that it
41:11
works properly. So it's it might be the same
41:13
question, but you are running three different prompts on three
41:15
different LMS, and those are kind of tested because every
41:19
time they update the learning, they update the version.
41:23
You figure out that certain things are not matching.
41:27
Some of the prompts work, simple prompts work, but some of the
41:30
prompts, you need to kind of change those because the output
41:34
may not be what you expect. Or sometimes suddenly you have
41:38
to add another rule. So I wouldn't see it.
41:41
You know the same front works across all.
41:43
We have to maintain like a prompt repository for different
41:49
things, yeah. Yeah, exactly.
41:51
You have to really look at how each LM likes to be spoken to.
41:55
They're very finicky in that way.
41:57
And so I I look at that and I think about it, yeah, the
42:02
basically the prompt engineering and the prompt design that you
42:05
have and the complexity that you add when you add a new model.
42:09
And it's almost like one thing that's becoming really clear to
42:14
me right now is how. Unless you absolutely have to.
42:19
You're adding so much more overhead by bringing on a new
42:24
model that it's better if you can.
42:27
If you're not like hitting a certain scale, and you don't
42:31
absolutely have to, then you don't need to go out and.
42:34
Use other models because you just have to start thinking
42:38
about all these new things like, OK, what?
42:41
When do we use this? What's the cost of it?
42:43
How's the prompt? What do they need to be looked
42:46
in like? And all of that is bringing on.
42:50
Just unneeded complexity if If yeah you're not at that scale.
42:55
Yeah means like. I would say that if you scale is
42:58
not huge, pick one model and get comfortable with it.
43:02
If you scale is usually you will end up with TPM and RPM and then
43:05
again at that point in time try to see if you can work with a
43:10
couple of models. Obviously we look out on the
43:13
lookout for you know, new models that you know.
43:15
You can see if I need to replace something in the near future,
43:20
but you know, if you're beyond that then you have to deal with,
43:24
you know, all of these issues I would say whenever you bring up
43:28
a new model. And then the other thing is, you
43:32
know, we didn't talk about latency right?
43:35
So you know, although these models work, depending upon how
43:40
much load is there on the platform, your results might
43:43
vary. Between I'm getting a result in
43:45
a second to I'm getting a result in 10 seconds.
43:49
To I'm getting a result in 15 seconds or after 15 seconds
43:52
couple making announcements. So that that is another thing
43:57
and that happens in real time, right, Like you, you don't have
44:01
control over it. So and things will improve
44:04
because there is a big shortage of GPUs, right.
44:07
Yeah. But as they improve, I think
44:10
these will be problems that, you know, people won't think about
44:15
like, you know, maybe two years, three years down the line.
44:18
But right now those are challenges, right?
44:20
As you kind of scale, yeah. It's so funny.
44:22
I was thinking about that when you were talking about the
44:24
prompting and how you need to have a special prompt for each
44:27
model and it's like and this does not feel like the future if
44:32
this. In Minority Report, I did not
44:34
see Tom Cruise having to pull his hair out because he had to
44:38
figure out the different prompts for each super smart AI.
44:43
You know, like that does not feel like the future at all and
44:46
hopefully we figure that out in some way, shape or form.
44:49
But the other thing that I was going to ask Ohh.
44:52
Along these lines are I'm sure you've done the cost benefit
44:56
analysis of buying a bunch of GPUs or just renting GPUs from
45:01
some of these GPU providers. What did that look like?
45:05
Because you're at this really interesting place where you've
45:08
got a tonne of scales, so you're hitting the rate limits on the
45:11
API's. But you said earlier that it
45:14
doesn't make sense for us to do the GPUs because those are
45:18
really expensive. So how did that whole cost
45:21
benefit analysis? Book.
45:23
And what made you ultimately opt for the APIs?
45:28
Yeah, so you know, I'd say that the cost benefit analysis that
45:32
anybody does, right, holds for that point in time.
45:34
So in the future this might change.
45:37
Right now it's see we are at a scale wherein you know getting
45:41
GPUs will be more costly because you know if I get GPS it can
45:45
handle more than what I need to, right?
45:48
But then for high availability and other things, I need to keep
45:52
those GPUs U and running, which is like an extra cost here I'm
45:56
basically paying on as needed basis, right?
45:59
And I believe that you know some of these LLM providers are
46:03
discounting costs, right? Like if you kind of look at the
46:06
amount of money that they're spending versus what they're
46:08
making, it will take some amount of time for them to actually
46:13
recuperate the cost, right. But you know they have enough
46:17
funding. Somebody like us, if I spend all
46:20
of that buying GPUs, then I'll be slow on like product
46:24
development and resources. So that's the reason I would say
46:27
and giving like a typical cost like if you buy a machine on
46:31
Amazon, like you know with good GPUs, yeah, like you know, I'd
46:35
say you know with good amount of RAM, 64GB and you know having
46:39
like 4GB cards or something like that, I can easily run up to
46:43
like $340 a day on a single machine right now you need a
46:49
cholesterol those. So and then you need to
46:52
basically like you know one is you get the GPUs, you deploy the
46:55
model that is 1 aspect even if you're not like fine tuning the
46:59
model, a lot of people talk about fine tuning.
47:01
But fine tuning I would say is not that easy and then you have
47:03
to do a lot of other fine tuning to actually see the results,
47:06
right. That's the reality.
47:07
Like forget about like Twitter and YouTube videos, but but then
47:13
the other cost that comes into picture that people don't kind
47:16
of think about is you need to have DevOps or data centre guys
47:21
managing that hardware, making sure that it is highly
47:24
available. If something goes down, you have
47:26
to basically bring it back up, right?
47:29
You want to manage all of that complexity right now, which is
47:31
additional cost, right? So one is machine putting the
47:34
model, making sure that it works and your engineers are kind of
47:37
focused on that and then you have to do the high
47:39
availability. So there is people, GPU
47:42
maintenance and management cost. And if something goes wrong, you
47:45
can only blame yourself. You have APIs, one model doesn't
47:49
work. You can switch immediately.
47:51
By the time you're that big that where you can do that other
47:55
piece. That is the time when you'll
47:57
basically go and say, OK, now I'll basically use my own data
48:01
centre and use my own GPUs, and I'll basically load up an open
48:04
source model, train it properly and then use it.
48:07
But you know, there's always a stage.
48:10
We are not at that stage right now, but you have to kind of
48:12
factor in all that costs, not only, you know deploying the
48:16
model and bringing it U, it's like maintaining it, updating
48:19
it, running it, making it highly overrated.
48:22
So the cost kind of goes up, the management and maintenance also
48:25
goes up. That is a wise way of looking at
48:28
it. I completely see your
48:31
perspective because we talk a lot about that especially.
48:35
With the ML 1.0, as some people call it, the build versus buy
48:40
decision is a very, very common question that comes into play.
48:44
And for a lot of it, it would be like, well, why do I need a
48:47
specific pipeline tool? I can just use this off the
48:50
shelf. Open source tool an I don't need
48:54
to. Have some managed service and
48:58
really in the beginning. You can get really far if you
49:04
just let someone else deal with those complexities and you deal
49:07
with your special sauce and you deal with as you said, it's
49:11
almost like you glossed over it. You're very much focused on the
49:15
product right now. You're not focused on the other
49:18
crap that's surround it, like networking, GPUs together or
49:22
fine tuning models that are open source models and who knows how
49:26
well that actually goes and getting the data, labelling the
49:29
data, cleaning the data, all that stuff that you have to do.
49:32
And all of a sudden you hire a few researchers, mill
49:34
researchers from Facebook, and boom, your costs skyrocket
49:38
again. Yeah.
49:40
And then you're burning money. Yeah.
49:42
And you're not delivering things to people, right.
49:44
Like, yeah, results, right. So I think there is always this
49:47
tussle. But again, as I said, right,
49:49
it's a point in time, right? Like over a period of time as
49:52
the things shift, then it becomes more viable to have your
49:55
own right. And in some cases it is a
49:58
requirement for an enterprise, for example, right.
50:01
We are not a starter, but for an enterprise we're just trying to
50:04
bring this in house as security is primary concern.
50:07
So they will start with that as the best thing.
50:11
I want it in house, then you can do the other stuff right.
50:14
So it all depends upon your requirements, your perspective.
50:17
But yeah, for a startup or anybody who's trying to use it
50:21
at say, you know, start with whatever is out there in the
50:24
market first, make sure that you are able to build and get what
50:27
you want. Then you can optimise, right?
50:31
Otherwise you're spending time and you don't even know whether
50:34
you'll get to what you want to get to in the first place.
50:38
Well put, well put. Well, last question I've got for
50:42
you is you've been knee deep in AI coding assistance and.
50:48
Learning how to. Interact with them.
50:54
And also make sure that they do what you want them to do.
50:57
What are some best practises that you've found when it comes
51:00
to debugging with your little code assistant next to you that
51:06
can help you along the way? I would say that whenever you're
51:11
using any kind of coding assistant or like even looms in
51:15
general right doesn't need to be like you know, some people use
51:17
chat GPD, some people can directly use anthropic chat and
51:22
so on. I would say first thing is
51:24
assume that element is there and you can answer generic questions
51:29
but it doesn't know everything and it doesn't know exactly what
51:32
you want. O as which context you can
51:35
provide. I think context is very
51:36
important, right? So if you're asking issue
51:39
related to or you know you got an exception and you're trying
51:42
to figure out how to fix it, you can paste the exception out
51:45
there and paste on the exception information that a human being
51:48
can also read. But most of the people have
51:49
forgotten. Because now you know it's very
51:51
easy. Like you know you put a debugger
51:52
and shows you what line it's going to basically go through it
51:56
and understand it and then explain you what that exception
51:59
means. And in most of the exceptions
52:01
you have, this line has an exception.
52:03
And then typically we used on the learning that it has.
52:07
It will say this might have caused this.
52:08
Most of the times it is 7075 percent, 80 percentile.
52:12
In some cases it does not there. That is because it has generic
52:16
context, not the context. Which is your context, your
52:20
code. O if you have like you know,
52:24
anything that you can provide for it to give you a better
52:27
answer, provide that as a context.
52:29
Like you know, hey, this is the file.
52:31
This is the code that I was running.
52:33
I'm seeing an exception in this particular code.
52:36
Here's the code, right? And this is the exception that
52:39
I'm getting. And this exception is happening
52:41
when so and so things happen. Alright, then you will get
52:45
better results. Otherwise you're going to get
52:49
like results which you know may hit the mark or maybe totally
52:52
off the mark, right. But in case of exceptions and
52:55
the other pieces, most of the information is out there.
52:57
So you know, most of the times 80 percent, 85% is out there and
53:01
you'll be able to solve the issue.
53:03
But for complicated cases where any apps, services which are
53:06
running and things can happen in a runtime when you're seeing an
53:09
exception, providing that additional context always helps.
53:14
And if you don't have the context, then it can
53:16
hallucinate. It can do a bunch of things.
53:18
Even sometimes with context it can hallucinate.
53:19
If your question is not right, another way to put it is
53:24
experienced developers who know how to code what is written and
53:30
the overall thing would be able to derive more better results
53:33
than people who don't know anything right?
53:35
Because they now know like you need to know something to ask
53:39
the question right? Like hey let me describe this is
53:42
the situation that I'm in and this is what I'm seeing.
53:44
What do you think is the issue versus I'm seeing this issue,
53:48
what do you think is the issue, right.
53:51
So that's what I will say is very important.
53:54
So although we say that a non programmer can become a
53:57
programmer to certain extent for simple examples, but if you're
54:01
doing complicated stuff, the guy who's asking the question needs
54:05
to make sure that he has to logically provide the context
54:09
that can actually help answer the question in another way,
54:13
right? So yeah.
54:14
So sometimes I call it oxymoronic, but yeah.
54:18
Well, nothing about this shit makes any sense to any of us, so
54:23
I like how that you've been able to find a few patterns and
54:26
you've been kind enough to share them with us.
54:29
This is awesome. And for anybody that is
54:33
interested and wants to try out Bittu, we'll leave the link in
54:37
the description and I think that's it for now.
54:41
Thank you Dimitris. This was great conversation, to
54:44
be very frank.