Latest / Agentic Conversations (formally mlops.community) / Building Cody, an Open Source AI Coding Assistant // Beyang Liu // #173

Transcript

Building Cody, an Open Source AI Coding Assistant // Beyang Liu // #173 · Agentic Conversations (formally mlops.community)

0:00
My name is Biang Liu. I'm the CTO and cofounder of
0:04
Sourcegraph and I drink my coffee black.
0:07
Hello and welcome everyone to the ML OPS Community Podcast.
0:11
I am your host Dimitrios and I'm flying solo today talking to Be
0:16
Young. Wow, what a conversation.
0:19
I loved. His idea of context is all you
0:23
need. That is all I'm going to say
0:26
because I do not want to give away too much.
0:29
He's been working his source graph.
0:31
For the last 10 years, building it out, absolutely crushing it.
0:35
And recently they released Cody, which you may have seen around
0:39
different code bases or you may have actually played with it in
0:42
your own code base. It's this way to query and level
0:48
up your abilities. Of understanding the code that
0:53
is in your code base. I want to give him a huge thank
0:58
you because this conversation was just spot on.
1:02
They brought large language model capabilities into Kodi and
1:07
he was an open book about how they've been using it, what the
1:11
hard parts have been, his vision for what it could be in the
1:15
future, and even what a developer could be in the
1:18
future. I loved the little roads and
1:21
side quests we went down today in this conversation.
1:24
Let me know if you like it. I liking and subscribing and of
1:28
course leaving a review. It would mean the world to me.
1:32
Share this with a friend if you're feeling especially nice.
1:38
And without further ado, let's get into this conversation.
1:48
The young you started Source Graph 10 years ago, you told me.
1:51
What were you doing before that? Prior to that I was a software
1:56
engineer at Palantir Technologies, which I think most
1:59
people have heard of now, but they do large scale data
2:02
analysis for government and large corporations.
2:05
Yeah, and so then what gave you the inspiration to do Source
2:08
Graph? The inspiration was, I think,
2:11
twofold. One was, as with many developer
2:15
tools, I think part of it was just scratching my own itch.
2:18
When I took a step back and thought about like the
2:20
day-to-day pain points of software development, I realised
2:24
that I was spending an inordinate amount of time trying
2:27
to read and understand and make sense of the code that already
2:30
existed and just like figure out heads or tails what was going on
2:34
there. So I really wanted a tool to to
2:38
alleviate that pain. And the second thing was
2:40
actually our experience at at Palantir.
2:43
So I I met and worked with my cofounder Quinn there and we
2:48
were on a team which was essentially drop shipping into
2:51
large enterprise code bases and trying to build software
2:54
effectively in that environment. And that was really an eye
2:57
opening because it was as you can imagine extremely large, old
3:02
and complex code bases and the the pain points even greater for
3:07
the software engineers that we were collaborating with on the
3:09
customer side. You must have seen every type of
3:14
code base imaginable. I can only do you have more
3:17
stories from those days. Yeah, definitely.
3:20
So we were working with a lot of kind of like financial services
3:23
and banking clients at the time and what was mind blowing to us
3:27
is some of these when you think of like software development
3:32
pains I think most people think of like you know like you had to
3:34
get merge conflict and I have to go through and resolve a bunch
3:37
of things we encounter teams are literally emailing blobs of code
3:42
back and forth because they're kind of satisfied with
3:44
compliance requirements. So like they would e-mail
3:47
lawyers like, hey, here's the code change that want to make
3:49
and you look this over and it was just crazy to us that in
3:53
2011 that was still happening. Yeah, I've heard about data
3:58
science teams. Doing all of their experiment
4:01
tracking in Excel files and then once they've figure out the.
4:07
Model that they want, they just send the Excel file to the
4:12
DevOps team and say like now it's your problem.
4:15
So it kind of feels like that was your life back then.
4:19
Yeah, pretty much. I think as much as we developers
4:21
like to gripe about, you know, more modern tools, the reality
4:25
is that it could be much, much, much worse.
4:28
Yeah, and so then you you thought, alright well I am
4:32
struggling when it comes to understanding these code bases
4:36
and seeing the developer productivity just wane and.
4:42
You decided why not make a tool out of it and I guess the
4:46
question that I have is. How were you so confident that a
4:51
technological tool was the solution?
4:53
Because it feels like it's almost like you need a part tech
4:58
and part education for the coders.
5:02
Yeah, definitely. So I I think when I took a step
5:05
back and looked at what was taking up time day-to-day, it it
5:09
was largely this this thing of like, hey, I need to go make
5:12
some change to this existing code base.
5:14
But before I can actually get started writing the feature, I
5:16
have to go and understand a bunch of different pieces that
5:19
already exist and how they fit together and how they're related
5:22
to the change that I want to make.
5:25
And I I had some points of inspiration for how I might go
5:29
about solving that. So I had previously, you know,
5:33
done a brief stint at Google. And Google has an amazing
5:36
internal development ecosystem and sort of like the
5:40
centrepiece, the shining star of that ecosystem was this tool
5:43
called Google Code Search. And the what Google Code Search
5:49
provided was the ability to search across the entire mono
5:52
repo inside Google and navigate the code.
5:55
So you didn't have to set up any kind of like local development
5:58
environment to be able to jump to definition or find
6:00
references. And to me that was sort of like
6:02
the bread and butter of what I did day-to-day as as an
6:05
engineer. And so I started thinking about
6:08
how could I construct a tool like that, but outside the
6:12
confines of Google. And in the meantime, my
6:15
cofounder had also experienced some similar tools in open
6:19
source. So there's a number of open
6:21
source code search engines that people have created and used
6:24
over the years in different communities, Open Grock and
6:28
Hound and and things like that. And so that was kind of like the
6:33
rough picture in our minds, like hey, those tools were useful.
6:35
How can we build that and make that kind of accessible to every
6:39
developer? And so then that was 10 years
6:42
ago right now, how have you seen?
6:45
That landscape change over the past 10 years, if at all.
6:49
I imagine there's still banks that are emailing around code
6:53
snippets because of compliance issues, but what?
6:57
What do you feel like the evolution has been?
7:00
Yeah, I think the the development world has come a
7:02
long way. I mean a lot of those banking
7:05
code bases were kind of a point of inspiration from a pain point
7:09
point of view and and now we're fortunate enough to have a lot
7:11
of those banks this as our customers.
7:14
Turns out, you know, we were solving a problem that was not
7:17
just inspired by the pain points we encountered.
7:19
It actually ended up yielding a solution that was was useful to
7:23
a lot of those organisations. But to your point earlier, I
7:27
think one of the things we underestimated was just how much
7:31
education it would take to educate to the kind of like
7:34
developer mass market about the usefulness of a tool like Code
7:38
Search. So in the early days, even
7:40
through till now, I think that the developers that like best
7:46
grock the utility of code search are still folks like people who
7:50
did a stint at Google or people who did a stint at Dropbox or
7:54
Meta. These are places where they felt
7:56
the pain point so severely. They built their own internal
7:59
code search engine and then it became one of the tools that
8:01
people rely on day-to-day. But then a lot of developers, if
8:04
you haven't worked in an organisation like that, you
8:07
don't have any strong priors for what code search is.
8:09
You're like, oh, is it like Control F in my editor or is it
8:12
like grep on the command line? And it doesn't really click with
8:15
people in a short amount of time.
8:18
And that remains to be kind of like a big, I guess it would
8:22
technically fall under the umbrella umbrella of, you know,
8:25
developer advocacy or or marketing.
8:27
For us, there's this issue of educating the vast majority of
8:32
developers about how to use this tool and why it's useful to
8:34
them. Yeah, I it's.
8:39
That jumped out at me right away is how you need to have this
8:45
tool, Needs to land with someone who understands the value of it.
8:50
Yeah, it needs to be something that people have felt that pain
8:55
and they realise there's got to be a better way.
8:58
Because I'm pretty sure my friend at XYZ company he's not
9:03
spending or she's not spending 2 hours a day.
9:06
Just trying to understand what this 5 lines of code.
9:10
Actually does and it's poorly documented.
9:13
Or the OR the documentation that is there says do not change
9:18
this. Whatever you do, it's like wait,
9:21
why? Why can't I change this?
9:23
And you don't want to know. Just don't do it so that that
9:28
feels like something that you are trying to fix now.
9:33
You set out to build a code search engine.
9:37
I I mean search in 2010 feels like it was in 2011.
9:42
It was a whole different beast back then.
9:44
How has your infrastructure and what you're doing around search
9:50
changed and especially just in like the last two years, because
9:54
with all of the advances I would say in in search in the last two
9:58
years. You have to have.
10:01
Done a lot of upgrading, I would imagine.
10:04
Yeah, definitely so. So the domain of search has
10:07
changed a lot in the past ten years and specifically in code
10:09
search. One of the things that we
10:11
started out building and prioritising that I think was a
10:14
step forward in terms of tools that were publicly available was
10:18
we viewed it not just as a strip search problem per se, but also,
10:23
you know, what do you do after you click into a result?
10:26
We're trying to understand a piece of code.
10:28
You click on that result and then you're staring at a code
10:31
file and a lot of what you need to do is do things like the
10:35
definition or find references. You're essentially walking the
10:39
reference graph of all the different components that fit
10:41
together. You're trying to understand what
10:43
those nodes are and what the edges are in terms of how
10:46
different parts of code relate to one another.
10:48
And we started out by focusing on constructing that code graph.
10:53
So essentially compiling the universe and extracting the
10:57
symbol tables so that we could execute a jump to definition or
11:01
a find references request without the user having to go
11:04
through the rigmarole of you know getting the the code to
11:08
build, which is an exercise in itself.
11:13
And so I think like the first big advancement we created was
11:16
this like a graph concept where essentially like mapping and and
11:20
graphing all the code in the world and feeding that into our
11:23
product as both a code of exploration tool and a signal
11:26
into the search side of things. More recently we've been looking
11:32
at incorporating AI based signals into the search results
11:37
and the search relevancy. I actually had a background in
11:40
machine learning way back in the day.
11:42
So I did my undergrad thesis in computer vision, but that was
11:45
like one or two AI revolutions ago.
11:47
And so back then, the quality of the models wasn't anywhere near
11:52
to where we thought would be useful in production.
11:54
But in the past couple of years, things have just come so far so
11:59
fast. And we're now in the process of
12:01
incorporating large language models into many different parts
12:04
of our product, both in kind of the generative sense of like,
12:07
hey, can I ask a question and get an answer or can I get it to
12:10
generate some code as well as baking kind of signals informed
12:14
by models into the ranking of the the code search engine
12:19
itself. What do you mean by that second
12:22
use case? Yeah, so for for a long time the
12:27
the the main interface to source graph Code search was supporting
12:32
string literal and regular expression searches.
12:34
So the idea is that you type in a partial string match or a
12:37
regular expression describing the pattern you were looking
12:41
for. And we supported that at both
12:43
kind of like a textural level and a symbolic level.
12:45
So you could actually type in a regex that matched like a symbol
12:48
name, a function name that you're looking for.
12:50
And we were smart enough to extract only the function names
12:53
and eliminate matches that were not symbolic.
12:57
But more recently we've been trying to build a query
13:00
interface that is more accepting of kind of like natural language
13:04
queries. And there's two types of natural
13:06
language queries. There's kind of like a fuzzy
13:08
kind of almost keyword style search that's similar to what
13:11
you might type into Google. When you search for something in
13:13
Google, you don't necessarily articulate it in grammatical
13:17
terms. You just kind of like blurt out
13:18
the first couple of of half formed words that come to mind
13:22
and somehow like the result you're looking for is almost
13:24
always at the top. And then the second form of
13:28
natural language is, is when you actually do articulate it in
13:30
grammatical terms and we want to support both types.
13:33
You know the former because a lot of queries are just like,
13:35
you know, like SAML, SSO, whatever.
13:38
I'm envisioning like a function in code.
13:41
I wanna jump straight to it, type out like your first half
13:45
formed thoughts and then we get you instantly there.
13:48
And then the second part is when you are trying to articulate
13:50
precisely like, hey, I'm looking for the thing that connects to
13:53
this thing and that thing. And so you kind of write out a
13:55
complete sentence describe what you're looking for.
13:58
And we want to support both. In order to support those types
14:01
of queries, have to incorporate AI in into the the stack in some
14:06
way, because otherwise you just wouldn't be able to do that kind
14:09
of like level of fuzzy matching. Yeah, it seems like on the one
14:14
hand, if you're just doing the Google search style, then the
14:18
AI. I'm not sure where it comes in.
14:21
I understand on the other side where yeah, if you're writing
14:25
almost like a prompt. And you're prompting it to say,
14:28
hey, where is this? What is it I wanna see, I wanna
14:31
know what's going on. And you're a senior developer,
14:34
so tell me in a nice way. Yeah.
14:37
So in in the keyword style search, where it comes into play
14:40
is, I think as you know and as most of your listeners know,
14:44
there's these things called embeddings, that there are the
14:46
internal representation of language models.
14:48
It's how the language model represents a string of text.
14:52
And those turn out to be pretty decent, decently useful for
14:57
searching for stuff. So basically you you map the
15:00
different functions in your code base to embedding space.
15:03
So you're essentially generating these large numerical vectors
15:06
that represent in some sense the semantics of that piece of code.
15:11
And then you have the user query which you also map to that same
15:14
space. And so you have these two
15:15
vectors, 1 representing the user query and one representing A
15:18
snippet of code. And then you do kind of like a
15:21
dot product between them which tells you the distance they are
15:24
between each other in semantic space.
15:26
And then when you're doing a search, it's essentially like a
15:29
nearest neighbour match between those two points in embedding
15:33
space. And so would that allows us to
15:35
do is it permits us to reveal you know like fuzzier level
15:39
results. So in standard keyword search
15:42
you kind of have to still get like bits and pieces of the
15:45
actual keywords. So I'm looking for like I don't
15:49
know, I guess to use the auth example again because they've
15:53
got auth recently. So IDC is like an auth protocol.
15:57
And so if I'm looking for the handler that supports NHDC and
16:00
my code base with keyword search, I would probably have to
16:03
type in OIDC or Open ID Connect and that would be at least like
16:07
a partial match against what I'm looking for.
16:09
But with the more embedding style search, you could type in
16:12
SSO handler or SSO handler based on Oauth or something like that.
16:16
And those terms oftentimes are close stuff in semantic space
16:21
and embedding space to yield a search result that you otherwise
16:24
would not have gotten with just standard keyword search.
16:28
You guys must have been using vector databases for a while
16:32
now. I get the feeling like you were
16:34
doing these embeddings before they were cool as my wrong to
16:37
say that I think we were a little bit ahead of the curve.
16:41
So we started looking into these signals probably late 2021,
16:46
early 2022. So we got a head start by a
16:48
little bit, but I would say look like we're not experts in vector
16:53
databases per se, because a lot of the nearest neighbour
16:56
searches that we do are actually still within the scope of a
17:01
single repository. And I think what a lot of people
17:05
don't realise is that's on the order of like millions of
17:08
results. And if you're dealing with that
17:11
magnitude of results then a for loop works pretty well.
17:15
You're literally just like scan through all the things in your
17:18
index and and dot product is is pretty cheap and so you can do
17:22
that and and it's pretty fast. Then I I think we will start to
17:27
get more into vector databases in the coming months when we
17:31
start thinking about scaling this nearest neighbour search to
17:34
a larger corpus of code beyond the scope of your current
17:38
repository. OK.
17:40
I like that. And know the other piece that I
17:42
was thinking about as you were talking through this is, can you
17:47
break down what the actual tech is behind this because it feels
17:51
like. You I'm not sure if over the
17:56
last, whatever 10 years that we've had to completely replace
18:00
how you're doing things or if you've been able to bolt on
18:03
these new features on top of it. And So what is, what does it
18:08
look like? Yeah, so there's, it's been a
18:11
combination of both. So we have like a completely
18:13
brand new product, which is Cody, our AI coding assistant,
18:16
which we basically built from the ground up.
18:20
But also we've been able to incorporate a lot of the AI
18:23
signals into our existing code search engine without rewriting
18:26
that from the ground UO. Our current code search is still
18:31
primarily keyword based, but it's probably going to evolve
18:34
into a system that has kind of a hybrid back end.
18:38
So we're using both keyword search and embeddings based
18:42
search as underlying search providers.
18:46
So both of those will go and fetch like an overset or a super
18:50
set of results that might be relevant to the user's query or
18:54
request. And then we'll have an
18:56
intermediate layer that's also AI enabled that ranks the
19:01
results from both to bubble up the most relevant results to the
19:05
top. So happy to chat about either.
19:07
I think both are extremely interesting problems, both the
19:11
challenge of incorporating AI signals into an existing product
19:14
as well as developing a new product from the ground up that
19:18
takes advantage of what elems are capable of of these days.
19:21
I feel like, let's start with incorporating AI into a product
19:26
that you already had, because I get the feeling a lot of people
19:28
are in that position and I like to make the jokes that most.
19:35
Product managers are going crazy right now.
19:38
Rd maps have been completely thrown out the window and
19:41
everybody's asking them what their AI plan is or if we can
19:44
add AI to this. Yeah, so I feel for them.
19:49
This one goes out to all those product managers but.
19:52
What did it look like for you? Like, how did you decide to
19:55
incorporate it in? And then how did you actually
19:57
make it happen? Yeah.
19:59
So, you know, how do you incorporate AI into your
20:03
existing product without upending your entire
20:05
organisation, especially when probably your CEO is breathing
20:08
down your neck and saying, you know, hey, what's our AI
20:10
strategy? What's our AI strategy?
20:12
I think that's a pattern that we've witnessed across the
20:16
board. And to some extent, you know,
20:18
we're experiencing that ourselves.
20:20
You know, I think we're in this weird position right now.
20:22
Where? Everyone, almost everyone sees
20:27
the potential of these technologies, but not everyone
20:30
has thought through precisely how they can be made useful in
20:34
their particular domain area. And so we're in this weird
20:38
position of there being just like a lot of top down pressure
20:40
to adopt AI in in some sense, but with not a lot of clarity on
20:48
the ground in terms of how to incorporate it.
20:51
And usually it's the opposite, right?
20:52
Like, you know you're you're in the weeds of the product and
20:55
you're kind of iterating against user feedback and you have these
20:57
insights that kind of like bubble up to these top level
20:59
insights. Whereas now we're trying to go
21:02
with like the top level mandate of like, hey embrace AI and then
21:06
it's like, oh how so I I don't know if like I'm, I'm an expert
21:12
on this matter, but I I I think you know in in when when we come
21:16
to trying to like grapple with with this task.
21:18
I think what we found works best is you always got to start with
21:23
something like simple I think and something that's within the
21:26
bounds of your comprehension. You don't want to go and try to
21:29
like take on the big, you know, uh dreamy task head on like you
21:36
don't want to go and try to start building AGI before you
21:38
have kind of like basic primitives in place.
21:41
And so for us, I think we, we started with, you know from the
21:46
point of view of improving our existing product.
21:48
We started with the question of like OK, what are the
21:53
capabilities of large language models there?
21:56
There's a couple of interesting applications that seem to have
21:59
emerged. One is code generation, another
22:02
one is kind of question answering.
22:04
And then a third one is this kind of embeddings based search
22:08
where you can have kind of this like more fuzzy or semantically
22:12
informed information retrieval mechanism.
22:16
And So what are the different products or sorry, what are the
22:19
different parts of our product that could take advantage of
22:23
those abilities? And then let's start by kind of
22:26
like rolling in some of these signals in a way that improves
22:29
upon the baseline. So the first thing we did was
22:32
take an off the shelf embeddings model and do nearest neighbour
22:37
search on that and see what kind of results we were able to get
22:40
to augment our existing kind of like trigram regex based search.
22:46
And the initial results were pretty poor overall, but with
22:51
like some glimmers of promise. You know, like 95% of the time
22:56
you're like, why did it think that these two were, you know,
23:00
this this symbol was like close to the user query that's
23:03
complete garbage. But then there was like in that
23:05
5% of the time where like ohh, my gosh, it actually found two
23:08
things that were semantically pretty close but had no keyword
23:11
overlap. And so that was the initial
23:13
thing and we iterate on that and started to to fine tune the
23:18
embeddings model a bit and and iterate on improving that.
23:21
And that was kind of how we did our toes in the waters
23:24
initially. And I get the feeling you had to
23:28
have been looking at all different types of use cases,
23:32
and one that I think about especially for.
23:35
What you're doing is, hey, let me highlight this.
23:40
Group of code or let me just like ask of this repository what
23:45
the hell is going on here, right?
23:47
Getting an answer back, right? What made you not go with that?
23:52
Or maybe that's something that is on the road map.
23:56
Yeah, so that's something that we we support now in the form of
23:59
Cody, which is our AI enabled editor editor extension.
24:04
So you can ask Cody a question about your code base.
24:07
It will do a bunch of searches in the back end and then answer
24:11
your question as best it can based on the context of your
24:13
code base. But in the beginning we didn't
24:15
do that because when we started investigating large language
24:19
models, this was before CHPT was released.
24:23
So GPT 3 was kind of like the most advanced, generally
24:27
available model at that point. And I think what we and a lot of
24:32
people discovered is, you know, GD three, So it had kind of all
24:37
the, the, the basic capabilities that were later found to be
24:41
present in in ChatGPT. But the amount of prompting that
24:46
it took to unlock those capabilities, you basically
24:49
couldn't unlock it reliably enough.
24:52
And so with just the base model of GPD 3, you couldn't just like
24:57
place relevant snippets of code into context and then say like
24:59
hey, can you answer the user's question based on these existing
25:04
snippets? It would just go off and like
25:06
you know try to complete some other piece of text that I
25:08
thought was relevant. What changed that was the
25:12
release of Chat CBT when the fine tuning to make it like
25:19
better at question answering and also um the kind of logical
25:26
synthesis capability to the extent that you know Elms can
25:29
reason that made it a much better model for this use case
25:35
of like hey you know what the heck is going on in my code.
25:37
Can you explain it to me? Yeah.
25:39
So breakdown, Cody then because I.
25:43
Guess I was under the impression that it.
25:46
It didn't have all those capabilities, but it seems like
25:50
I was totally off. Give me the breakdown on what
25:53
code he is because it's it's like a chat with your data type,
25:58
bot that or QA where your data is not like a notion document,
26:04
it is your repository. Yeah definitely.
26:07
So Cody has a couple of main interfaces, so the two main ones
26:16
are one is inline autocomplete. So as you're coding, it will
26:20
auto complete the code that you're trying to type based on
26:25
the context of your code base. So it fetches contacts from
26:30
different parts of your code base, tries to take hints from
26:33
signals in the code that you're writing to determine what other
26:37
snippets of code might be relevant to pattern match
26:40
against when it's completing the code you're trying to write.
26:45
The second interface is more of like an explicit prompting
26:48
interface. It's it's more akin to chat,
26:51
although we don't like to say the word chat because I think
26:55
open-ended chat is actually not a great way to present the
26:57
capabilities of the language model in the context of coding.
27:00
So our chat, so to speak, is more like a natural language.
27:04
Ripple of sorts, where there's commands that represent common
27:10
things that you want to do through explicit instruction.
27:13
Things like generate unit test or explain some hairy piece of
27:17
code or generate some documentation for a piece of
27:19
code. And those are things where you
27:22
would write out and explicit set of instructions like, hey, can
27:25
you explain this selection Or I'd like to generate a unit test
27:29
for this code and then you hit enter and then in the transcript
27:36
either inline in the code. So where you've made a
27:39
selection, there's kind of like a text box that will appear
27:41
there and you can have a conversation that's attached to
27:43
a specific range and code or you can have it just like general in
27:47
the sidebar. So think of it as kind of like a
27:50
terminal source, but it understands like human language.
27:54
You you enter in your query there and it will either respond
27:57
with an answer your question, or if you ask it to generate some
28:00
code, it will generate some code that follows the instructions
28:02
that you provided. There's so many cool things
28:05
about this. The big question is, does it?
28:10
Tell you. Why you can't delete that one
28:15
piece of code that is commented on?
28:17
We've never touched this code, so it can It can, provided that
28:24
the underlying context fetching mechanism discovers the relevant
28:30
pieces of code to answer that question.
28:32
And so one of the things that we think is unique and
28:36
differentiated about how Cody works is the content, sorry the
28:39
the context fetching mechanism. So we have this catch phrase
28:44
that we started to say, which is context is all you need, which
28:51
is kind of like a riff on the attention is all you need paper
28:54
title. But essentially what we found is
28:57
that the quality and accuracy of responses from Cody, whether
29:01
you're doing inline autocomplete or answering a question in the
29:06
more explicit Rebel interface, probably like 95% of the time
29:12
it's dictated by hey, did you fetch the relevant of or did you
29:17
fetch the relevant pieces of code into the context window of
29:21
the large language model? Because if I'm trying to answer
29:24
your question, so let's say your question is like, hey, why can't
29:27
I delete this piece of code. Now if underneath the hood we go
29:31
and find maybe we do like a fine references or go to definition
29:36
to understand like where that piece of code is being used
29:39
elsewhere in the code base and then that snippet of code shows
29:44
that the function is being invoked in a certain way.
29:47
Then using a language model like ChatGPT or Claud or an
29:53
equivalent chat based model. Those models are generally good
29:56
enough at doing kind of like some primitive form of logical
30:01
reasoning where I can say like, oh like here's a comment saying
30:03
like don't change this. And then here's an example in
30:27
the code of using that function in a particular way.
30:30
And then it's able to synthesise those two facts into an answer
30:33
explaining why that comment exists.
30:36
On the other hand, if your context fetching mechanism
30:39
misses one of the snippets of code, let's say like it can't
30:43
find any usage examples for whatever reason, then the model
30:45
will just more likely than not make up some BS answer as to why
30:50
why that is, you know? Yes, yeah.
30:54
And I I get the feeling it's like if only it was that easy
30:58
where it was that piece of code could be linked back to another
31:04
piece of code I feel like on those really hairy databases or
31:09
sorry, code bases. Yeah, what you get are like 5
31:15
different layers of code being dependent on other code and then
31:20
what you have, I don't know. I would love to see if the
31:24
search could pull up every piece of it and follow the bread
31:28
crumbs around. Yeah.
31:31
So the short answer is, today it doesn't because we haven't made
31:34
it good enough yet, but we're on the track to to getting there.
31:38
So essentially the intuition that we're following is, you
31:42
know, if you gave a human that task, like let's say you dumped
31:45
a human into this code base and said like, hey, we need to
31:47
change this code. Despite there being a comment
31:49
that says like do not change this code For these reasons.
31:52
You know there's some feature we need to build of you as like the
31:55
hapless human would have to go in like you would use that as
31:58
your starting point. But then you know you might
32:00
think of some searches that you might want to do like ohh like
32:03
related pieces of functionality to surface potential needles in
32:07
the haystack. Or you might do like a fine
32:09
references or a couple of go to definitions to do kind of like a
32:13
search around and and find surrounding code that's
32:16
relevant. And over time, you'd incorporate
32:18
those snippets into like a general picture of what's going
32:20
on in the code base until you have enough relevant snippets of
32:25
code paged into your working memory to make an attempt at
32:29
actually editing the code in a way that doesn't completely
32:32
break the system. And so that is essentially what
32:36
we're trying to make the AI do. I don't think like the AI is not
32:39
going to come up with some like magical insight.
32:43
It's not like you dump the entire code base in and just
32:46
magically pops out with the answer.
32:47
Like that's not how things work. Despite all the hype around AGI,
32:51
just these Helms and practise just aren't there yet.
32:57
And So what we are building is basically a system that can
33:04
replicate the process of how human understands and surfaces
33:09
relevant pieces of code in the code base.
33:11
And then over time like the first step is we have to have
33:15
Bible primitives. So like the one the the answers
33:17
that are answerable with like one hop.
33:20
You know like a fine references or go to definition request like
33:23
if if the answer is answerable based on the snippets you can
33:26
get from that one hop action. Those are the ones that we can
33:29
do a reasonable job of answering now and then Once that's
33:33
reliable enough, we'll do 2 hops and then the three hops and then
33:36
at some point we'll work our way up to maybe this thing that
33:39
feels kind of like semi autonomous in terms of how can
33:41
it, it can explore and discover things in the code base.
33:46
It just reminds me I heard a story back in the day and I
33:50
can't remember where I heard it, but it was something along the
33:54
lines of an engineer looking at a code base and reading the
33:58
comment and the comment said. Do not delete you stupid idiot.
34:04
Something like that. And the guy looked and it was
34:07
like. Man, that's pretty harsh.
34:10
Like that seems like some really aggressive language for this
34:15
simple function. And where's this function even
34:19
it's not useful. Let's like try and dig into it.
34:22
And they were digging into it for the longest time, and they
34:26
came to the conclusion that it wasn't actually being used
34:29
anywhere. And so they went ahead and they
34:32
said, you know what? Like whoever did this last was
34:36
exaggerating or. Since then the code base has
34:39
been updated. And you know, there's no more
34:41
dependencies on this function. And they created the function
34:44
and then it crashed the whole system and they had to rollback.
34:49
And from there they went and they dug up who wrote that
34:54
comment. Turns out it was the same
34:56
engineer who deleted it. It was like them talking to
35:01
themselves, but like three years back they were saying it and
35:07
they probably did that whole thing and went through that.
35:11
All cycle. And so they they had to.
35:17
Go through it again. So it's one of those things
35:19
where that's what Cody's for. It's one of those things and
35:23
look like I don't think the AI will ever get to the point where
35:25
it's like ohh like I just understand everything there is
35:29
about the world. So I know why this thing needs
35:32
to stay because of, you know, you know, I I'm generally
35:36
intelligent. I'm super, super intelligent and
35:38
by some magic I I just know the answer.
35:41
What it's going to do is just allow you to more rapidly
35:44
explore the channels and the pathways that you know that you
35:48
should be doing when you're trying to, like get a
35:50
comprehensive answer to a question that you have anyways.
35:53
But you you don't end up doing a lot of that now because you're
35:56
like, ah, like I've already sunk like so many hours into trying
36:00
to figure out this one thing. I haven't even written a single
36:03
line of code yet. It essentially like accelerating
36:06
and automating the toils and parts of the job so that we can
36:09
do a more thorough part of the job that humans are actually
36:14
good at, which is you know the kind of like high level like hey
36:17
I need to make this change to to the code base.
36:21
So. When you think about developer
36:24
productivity and as you're talking about.
36:28
Augmenting the developers abilities and being able to take
36:34
anybody on the street. Well, maybe not on the street,
36:38
but anybody that is in a software development.
36:41
Role and then. Cutting out all of that
36:45
cumbersome time that goes into the researching of figuring out
36:51
like what is actually going on here, how can I read this code?
36:55
It's much more. What I'm understanding is it's
36:57
much more than just. A copilot like experience where
37:03
you're building out autocomplete and OK, cool, I'm going to write
37:06
this function, I'm going to add this to it.
37:08
What you're doing is you're helping interpret, you're
37:11
helping answer Crest questions, you're helping find different
37:15
pieces in the code base that there's bread crumbs that have
37:18
been left behind. And so now as you think about
37:22
developer productivity, I know you mentioned before we hit
37:25
record that you feel like we're in this Gutenberg type moment.
37:29
Can you explain that a little more definitely?
37:32
Our our core thesis is that the the core of of being a software
37:37
creator a a programmer by today's standards is, is about
37:45
understanding code and then writing new code based on that
37:48
understanding. And so like that is that is the
37:51
thing around which everything evolves.
37:54
It's understanding source code and then outputting source code
37:56
to build features and functionality for the tool or
37:59
product that you're creating. And that should really be the
38:04
centrepiece of all the other tools that you use because
38:08
that's the core kind of like inner cycle of what you do
38:12
day-to-day as a programmer. Your job is kind of going
38:16
through this loop again and again of like understanding,
38:19
synthesising the code into your kind of worldview and then
38:23
making an update to that worldview by editing the code.
38:27
And so that's the tool that we're building.
38:29
We really think that this is a tool that will be kind of like
38:31
the centrepiece of most developers workflows.
38:34
Now to your point, this kind of like Gutenberg moment, I think
38:38
there's this question of like what is a software developer in
38:42
like 5 years or 10 years in in the world of a I.
38:45
And we actually think that there's there's a huge
38:48
opportunity in front of us to kind of expand the franchise, so
38:51
to speak of the set of people in the world that are able to
38:55
create software. If you think about it, there's
38:58
kind of like this tiny, like priestly cast of people that you
39:01
know can speak the language of of computers right now and and
39:07
we call them programmers or software engineers, and 99% of
39:11
people aren't able to do that. So the way they interact with
39:14
computers, the way they're able to leverage the capabilities of
39:17
computers to improve their daily lives, is limited by the
39:20
software that's created by that, kind of like tiny priestly cast.
39:26
And So what happens in a world where we give that power of
39:31
software creation, the ability to customise behaviour and
39:34
implement your own functionality, implement your
39:36
own automation to the remaining 99% of the people in the world?
39:40
And I think that's definitely within the realm of possibility.
39:42
Now, given the the advent of large language models, large
39:46
language models are potentially like the world's best mechanical
39:50
translators. They can translate from English
39:54
to German or to whatever human language you prefer.
39:59
And they're also great at translating natural human
40:03
language into programming languages as well.
40:05
And so I think in in five to 10 years, we'll think about a much
40:09
broader set of people being involved in the process of
40:11
creating software. And we'll need kind of like an
40:15
entirely different way of thinking about how that process
40:18
goes and what sorts of people are making contributions to this
40:22
kind of like code base. Yeah, because if it's as easy as
40:29
making an edit to Wikipedia, it's a whole different playing
40:33
field, right? Yeah, yeah.
40:36
And so I I also, I like what you said how it's almost like
40:40
there's this elite class that understands what's going on when
40:44
they look at GitHub and they are able to speak this language.
40:50
And so if you're able, if you can bring that to the masses,
40:53
what is that going to do for the products that we create?
40:59
Yeah, 100%. And we, we need tools that allow
41:02
this translation to happen reliably.
41:04
So I think, you know, today's world, it shows a lot of
41:07
promise. But we're still not quite at the
41:09
point where I can hand, let's say like my dad, a computer,
41:14
he's not a programmer. He has no background in
41:17
programming training. We're still not at the point
41:19
where he can go and basically talk to the machine and get it
41:21
to implement some piece of automation.
41:24
There's still like a little bit of debugging knowledge that you
41:26
need to have a little bit of ability to kind of like manually
41:29
adjust. The the code that AI emits, but
41:34
I think by building a system that has both LMS and this kind
41:39
of notion of like checking correctness and then that's one
41:42
of the things that source graph has traditionally been good at
41:45
because we have kind of like a compiler level understanding of
41:47
the code. You you can, you could envision
41:51
a system that is able to kind of like iterate and self correct on
41:54
itself and reliably generate code that edits an existing
41:58
piece of software based on some piece of natural language
42:01
instruction. And so that would be really cool
42:03
because then I could say like dad like you don't like, you're
42:07
missing some feature in Google search or some other piece of
42:11
software. You use a go and talk to your
42:13
computer, Describe what you want and it will actually like.
42:16
Figure out how to do the thing that you're trying to describe
42:19
to it. Well.
42:20
So here's where I find it funny because watching the different
42:27
YouTube videos that talk about how you can leverage ellms to
42:32
code now or nobody needs to actually know how to programme
42:36
because you can use LM's and that's all you need type thing.
42:40
Yeah. But then they proceed to go into
42:43
OK And then I just, you know, I create this regex expression and
42:47
you're like, wait a minute, how did you know how to create that
42:51
red? How did you know how to ask the
42:52
LM for that specific thing? And it's like you wouldn't know
42:57
how to ask for these things or how to phrase your prompt unless
43:02
you were actually a programmer. So it very much in, in my eyes
43:06
is still at that clunky phase where yeah, it's cool if you
43:10
know what's going on, but if you don't know what's going on,
43:13
you're still kind of lol. Yeah, we're still in the early
43:17
days and right now the the media applications that are actually
43:20
production sizable are still mostly for professional software
43:24
engineers. But I think it's kind of like
43:26
those old cooking shows. If you remember growing up like
43:29
you know, the person would do all the prep work and show you
43:32
how to chop the vegetables and put everything into a bowl and
43:35
they put something in the oven and then like you know, take out
43:38
immediately above it like hey, here's the fully formed souffle
43:41
or whatever. And then you would do the same
43:43
thing and yours would come out looking nothing like that and
43:45
you're like OK, there was some steps that.
43:47
Were omitted in in in this process.
43:49
And I think right right now humans still have to fill in
43:52
those gaps, and those humans still have to understand they
43:57
have to be literate with programming languages.
43:59
But I think in a couple of years we'll get to the point where we
44:01
have systems that are reliable enough that can do kind of like
44:04
the automatic checking for you, so that people who don't
44:09
consider themselves professional programmers as software
44:11
engineers can actually start being more reliable contributors
44:15
to code bases. Well, So what do you think then
44:22
is the argument for why that hasn't happened in the low code,
44:27
no code movement? Because that's been around for a
44:30
long time, right? And yeah, I know.
44:33
I guess whenever I've used low code, no code, it's kind of like
44:37
that. Same thing where I get
44:40
overloaded with complexity, or I still feel like you have to have
44:43
a bit of a systems mentality or a systems way of thinking about
44:48
it and a tonne of patience to learn how the no code tool
44:54
works. Yeah.
44:56
OI think traditional low code no Code tools were an attempt at
45:01
solving the complexity of software creation through the
45:04
introduction of abstractions. And that's indeed how most
45:08
people have tried to pare down the complexity of software
45:11
development. In any capacity, right?
45:14
Like introduce an abstraction layer that hides some of the
45:17
messy detail, prevents the user from shooting themselves in the
45:19
foot, and kind of like guides them along a happy path of
45:23
development. I think the the challenge with
45:26
with that approach is that there is a cost to the abstraction,
45:30
right? What you get is more guardrails,
45:33
but it comes at the cost of expressivity and this is
45:36
something that people have noted again and again with with tools
45:39
like that. It works as long as you're
45:41
trying to build the thing that they built in the demo, right.
45:43
But as soon as your requirements go beyond what the platform has
45:49
anticipated supporting, then you kind of hit a brick wall.
45:52
There's no way to get around it, or you have to.
45:55
You know, dive so much into the weeds that it would almost be
45:58
easier to ask a professional engineer just to write it in in
46:01
kind of like a a regular programming language.
46:04
I think language models have a potential to create the best of
46:08
both worlds, where maybe in the future when you're trying to
46:14
build an application and you're not a professional software
46:17
engineer, you don't understand how to interpret, you know, C or
46:20
Java or Python. You can just express.
46:26
What you want natural language just in the context of a regular
46:29
code base. So there's no like no code or
46:32
low code platform. There's no piece of abstraction
46:34
sitting between you and the code.
46:36
But what you're doing is you're articulate what you want in a
46:38
natural language. It's generating code for you.
46:41
It automatically cheques that it works and that it can kind of
46:44
explain back to you the code that it generated sort of
46:48
explaining the detailed things it's doing.
46:52
How does that relate back to your your high level piece of
46:54
instruction and so? I think that is, to me, a much
47:00
more compelling vision of the future, because then we can have
47:03
our cake and eat it too. We can have everyone be involved
47:08
in the process software creation without sacrificing any of the
47:11
expressivity or the power of a raw, Turing complete programming
47:17
language. Yeah, those guardrails that,
47:19
it's exactly that. Like you try and mess around
47:23
with it. It's a lot of the same
47:25
complaints that people have when they're using a managed service.
47:29
It's like the best part of a managed service is that it's
47:31
managed. The worst part of a managed
47:32
service is it's managed. So you can't get underneath the
47:37
hood when you really need to and you want to go.
47:40
And that maybe 80% of the time it's magical.
47:44
And then the 20% of the time you're like, I really wish I
47:47
could just turn a few knobs and change a few things on this.
47:52
Yep, absolutely. So when you think about how
47:57
you're building, especially Cody, I wonder if you
48:02
undoubtedly, you've gotten into the idea of what if we could
48:07
just make agents that would go and do what people want done
48:13
before they even have to go and do it themselves.
48:17
Yeah. Have you gone down that path?
48:21
Yeah, absolutely. I think the agents are all the
48:24
rage right now and I think there's a tonne of potential
48:26
there. I I think there there is a
48:29
future where we have a a bunch of these agents and I I think of
48:33
them more as. They're just computer systems,
48:37
right at the end of the day, where the input is some piece of
48:41
human instruction, likely natural language, and then a
48:44
series of actions that performs on behalf of the human,
48:49
involving it could be human in the loop or human out of the
48:53
loop. So, you know, either completely
48:54
automated or kind of like. Asking you for feedback along
48:58
the way. Um.
49:00
I think the world we live in today.
49:04
Well, I I guess I would, I would say like OK like that's the end
49:06
state, right. And the question is how do we
49:08
get there. And I think what you kind of
49:10
have to do is you have to start with kind of like the building
49:13
blocks, like what are the building blocks of agents.
49:15
So agents are these kind of like multi step things that happen in
49:19
response to a query or a prompt. In order to get to multi step,
49:24
we have to start with reliability at a single step of
49:27
that process and we're still at the stage where that single step
49:31
process is still fairly unreliable.
49:35
You know, a lot of people talk about language models being,
49:38
quote unquote, usefully wrong in in the sense that, like, you
49:41
know they will hallucinate. Yeah, from time to time, they
49:44
will mess up from time to time, but it's OK because there's
49:47
still more useful than not. Like even if it invents an API
49:50
that doesn't exist, you as a human can still be like, oh you
49:53
know that doesn't exist, but like the rest of the code you
49:56
wrote is still helpful. I can pattern match against
49:58
that. Once you start trying to compose
50:01
those into longer and longer chains of thought, the
50:04
reliability and the accuracy does tend to matter more.
50:07
Because think about each, each time, each step of the process
50:11
you're rolling, you're rolling the dice, so to speak, and if
50:15
you hit an error mode, then your thing will quickly go off the
50:18
rails. And so for A1 hop process, you
50:20
know, 70% reliability. Maybe that's tolerable because
50:23
it works well enough most of the time.
50:26
But once you're, you know, three or four steps into it, you know
50:29
what's you know, .7 to the. The 4th or 5th power at some
50:32
point decays to zero, Yeah. And so I think where we are with
50:35
agents right now is we're still, we're still at the point in
50:39
working with language models. Where the longer chains of
50:43
thought are just too unreliable to rely on for day-to-day use or
50:46
production use. I think we're we're maybe still
50:49
at least a year away from there. And what it's gonna take to get
50:53
more reliability. I think #1 is improvements to
50:55
the model layer, so training and fine tuning models to be more
51:00
agent. Friendly on agent friendly
51:03
exactly. Or like domain specific like if
51:05
you're training an agent to use APIs right?
51:07
Like there's a lot. There's a lot of low hanging
51:09
fruit I think left for enabling these to.
51:12
Be better users of APIs and to generate working code.
51:15
And then there's also figuring out how to compose them with
51:18
other systems, like things that might do static cheques or check
51:21
the correctness of the output and might fetch information or
51:26
context like Source Graph does for the agent to help improve
51:30
the accuracy. Yeah, it doesn't feel like
51:32
there's a clear stack for agents yet.
51:36
And there it does feel like, on the other hand, we're getting a
51:41
stack for using limbs, and that is becoming like vector
51:45
databases, the champion in this one.
51:47
And then you've got your, you've got your SDK libraries, and
51:51
maybe you've got some prompting or some explain ability or some
51:54
kind of tool to keep the guardrails on.
51:58
But and then of course the large language model is is there and
52:02
ever present. Yeah.
52:04
But when it comes to agents, I feel like what you just
52:09
described there hasn't been. Or maybe I just don't spend
52:14
enough time diving into the tooling around agents.
52:20
Yeah, but it there hasn't been that idea of like here's a clear
52:24
design pattern when you're trying to use agents.
52:28
Yeah, I think that's absolutely right.
52:30
I think that's largely because no one has figured out how to do
52:33
it reliably. And so there is no like example
52:36
of like hey like this works awesomely and it supports all
52:39
these things in production. At at at the point in which like
52:42
that becomes a thing, then I think there there will be like
52:45
at least a few open source projects or frameworks that
52:48
emerge where it's like OK, we can point to at least one
52:50
application that makes use of us that people use day-to-day or
52:53
they rely on in a production environment.
52:56
And I think right now it's mostly just ad hoc, like
52:59
everyone building with language models is kind of building their
53:02
own proto frameworks internally, because we're still experiencing
53:07
exactly what combination. Works well in practise.
53:12
You did mention one thing when it comes to dealing with agents
53:16
and how you feel like. There's a lot of possibility for
53:20
creating a fine-tuned model that just deals with APIs or API
53:25
creation. When you were building Cody, are
53:28
you using GPT on the back end? Are you using Claud?
53:32
Did you bring it all in house and are you using a open source
53:36
Lol. Yeah, So we we use LMS in a few
53:39
different places in Cody, so for the code generation and the chat
53:45
based. Generations of the Q&A.
53:49
We can use either Claud from Anthropic or GPD from Open AI.
53:56
We're also planning on supporting some of the newer
53:58
open source Lala based models there as well.
54:01
And then for the retrieval aspect of it, which is
54:04
generating embeddings and using those for search, we're using a
54:08
combo of Open AI embeddings and open source embeddings that
54:13
we've kind of fine-tuned for our purposes.
54:17
And so we are kind of getting into the model layer, um, and
54:21
we've been a little bit circumspect about investing
54:25
there because I think there's just.
54:27
There's a lot of good stuff available off the shelf
54:29
currently, and. In working with these things, we
54:33
think a lot of the low hanging fruit is still in the realm of
54:36
retrieval augmentation. So like how can we improve the
54:40
quality of the context that we feed into the models?
54:43
And so I don't think we've exhausted that yet.
54:45
And the fact of the matter is, you're just you're able to
54:48
iterate much more quickly when you're only dealing with model
54:51
inference and feeding in different pieces of context and
54:53
playing around with the context retrieval mechanisms, then
54:57
training a model or fine tuning a model because that involves a
55:00
whole cycle like gathering data. Meaning the data running, the
55:04
fine tuning, seeing the outputs. And so we probably will get
55:08
there eventually, but right now we're mostly a retrieval
55:11
augmentation shop. I always joke that fine tuning
55:15
is something that sounds so good when you say it and everybody is
55:19
just kind of like throwing it out there.
55:20
Like, yeah, our next step is we're going to fine tune a
55:23
model. And it's like, do you know what
55:24
goes into fine tuning? Have you really tried to do this
55:27
before? Because it's not like just
55:29
uploading your photo to Lexica and then getting like some cool
55:34
shit passed back to you 15 minutes later.
55:36
That's not fine tuning. As much as stable diffusion may
55:41
have made you think it is, that's not really what it is
55:44
with Lambs. Yeah.
55:46
Yeah, it's almost like a I I feel like I've seen it in some
55:49
capacities where it almost seems like a marketing, yeah, we're
55:52
like ohh, we're doing the real AI because we're we're fine
55:55
tuning and I think we will get to fine tuning at some point.
55:59
But we have this mindset, like I said in the very beginning, like
56:03
when you're getting into a I like my recommendation would be
56:06
start with the easy things first.
56:08
Start with the things that have the ability to add immediate
56:11
value. And so how are you dealing with
56:14
rate limits and followers or failovers?
56:19
Like, what do you do about that when all of a sudden Open AI or
56:25
Claude doesn't want to respond anymore and you're just like,
56:27
well, guess we can't call support?
56:30
Well, actually we we can't call support.
56:32
So we have good strong enterprise relationships with
56:37
both anthropic and open AI in place.
56:39
And so you're the lucky ones you are.
56:43
I suppose we are. Both of them have been very
56:46
helpful as we've we've kind of built out and scaled Cody up,
56:52
but the short answer is that it's just like another external
56:54
API. Like if if you're a product that
56:56
has any sort of external or cloud dependency, this is
57:00
something that you have to manage as part of your platform
57:02
and there is downtime at some points and that's something that
57:06
you have to convey to your customers that now and then
57:09
there might be downtime in service.
57:12
I think the the good news is that we're still in kind of like
57:15
the early days of AI. So a lot of folks do have kind
57:19
of like a general tolerance of like, hey, you know it it is
57:22
kind of like the wild, Wild West right now and and things are
57:24
moving quickly so. You know, if things go down for
57:28
a period of time or there's some turbulence along the way, then
57:30
that's just part of the journey. Well, I feel like you already
57:34
are doing a lot more than most people think about when you have
57:40
clawed and ChatGPT or just GT to hit her and so I guess you can
57:46
load balance a little bit that way.
57:50
Yeah. I think one of our philosophies
57:52
is we do want to support multiple language models because
57:57
you know, not just for kind of like the reliability and
57:59
robustness side of things, but you know as new models get
58:03
released, we wanna be able to incorporate the latest and
58:05
greatest into Cody. But you're not.
58:08
If I am playing with Cody, I'm not getting both of those
58:12
answers, am I? It's just hitting one of the
58:15
API's or are you? Because I imagine there's a lot
58:19
of roads we could go down when it comes to evaluation and how
58:24
you're evaluating which ones are better and.
58:27
Yeah. So right now you're getting
58:29
either one of those. We're not doing both.
58:32
We're not trying to like compare and contrast anything at flight
58:35
time. I think part of it is just it
58:37
was easier to implement that that way.
58:40
And then another part of it is also, I said before, you know
58:44
context is all you need and what we found is there are
58:47
differences between the two models in terms of what their
58:52
relative strengths. But by and large the the context
58:56
is what makes the difference. And so like the the the
59:00
differences in terms of like how good it is at generating code in
59:03
this language or that they largely diminish or even manage
59:09
once the context fetching quality is good enough in a
59:11
given language. Because what the model then does
59:14
is just pattern match against the the snippets that you feed
59:17
it, and so relies a lot less upon its kind of like.
59:21
Unique, memorised knowledge. Yeah, that's fascinating.
59:26
So it really gives it less roads to travel.
59:30
Or I almost look at it as your closing the possibilities that
59:38
it can answer back with you don't give it anything.
59:41
It's not like, hey answer me with anything that you've been
59:44
trained on. It's like very, very specific.
59:46
And the more specific that you can give it, the more
59:48
specificity that you can give it, the better it's going to be
59:54
and the more accurate it's going to be.
59:56
Yeah, exactly. I think like the rough analogy
59:59
is like, let's say you had two humans and they're both pretty
1:00:01
smart and one might be better at, you know, Python And the
1:00:05
other one might be better at Ruby.
1:00:06
But you give both of them access to, you know, a code search
1:00:10
engine or something like Google, and then you ask them to answer
1:00:15
a question. The answers are going to be a
1:00:17
lot closer. If they have access to the
1:00:18
search engine, then if you ask them to, just like you know,
1:00:22
write it from memory or from memory, yeah.
1:00:26
Well dude, this has been awesome.
1:00:28
I know we've went way overtime, so I want to thank you for being
1:00:33
incredibly open about all of this.
1:00:35
And for anybody out there that has not tried Cody, highly
1:00:38
encourage it, have some fun, go and play around with it, bring
1:00:42
it into your code base and see if it will tell you why you
1:00:46
can't delete that one fucking function that's been there since
1:00:50
the beginning of time. And and let us know too if you
1:00:54
end up trying it out. We have a discord that's
1:00:56
available and we love to hear feedback, especially the
1:00:59
critical feedback that allows us to to make the product better.
1:01:02
And thank you Dimitrios for having me on.
1:01:04
This was an awesome conversation.
1:01:06
Had a lot of fun. Yeah.
1:01:08
I mean, when it comes to feedback, I was thinking about
1:01:10
that. When you do start incorporating
1:01:12
Llama in there, I imagine that's going to be excellent.
1:01:15
If you can run them both at the same time and you can say, all
1:01:19
right, we've got Claude and we've got Luma.
1:01:21
Which one was better? Yeah.
1:01:23
And then you can start seeing like or you can start training
1:01:27
fine tuning. It's just as easy as a fine
1:01:30
tune. Dude, you're it's all good.
1:01:32
Yeah, you're Llama. Model, Yeah, yeah.
1:01:36
And ask them to rate each other too.
1:01:38
Yeah. Yeah.
1:01:39
Yeah, exactly. Anyway, that's it.
1:01:41
That's the broad. We'll end it there.
1:01:43
Thanks man. All right.
1:01:45
Cheers.