Latest / The Pragmatic Engineer / AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django)

Transcript

AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django) · The Pragmatic Engineer

• 0:00
  SPEAKER_0: Every programmer who works with these models the first time, it spits out like 20 lines of actually good code that solves your problem and does it faster than you would.
• 0:08
  SPEAKER_0: There's that moment when you're like, "Hang on a second, what am I even for?"
• 0:11
  SPEAKER_0: And then I tried this new feature of ChatGPT that they launched last year called Code Interpreter Mode, and I asked a question and it flawlessly answered it by composing the right SQL query, running that using the Python SQLite library and Spitting out the answer.
• 0:25
  SPEAKER_0: What am I even for?
• 0:26
  SPEAKER_0: I thought my life's purpose was to solve this problem that was a little bit existential dread.
• 0:31
  SPEAKER_0: It is scary when you think, "Okay, I earn a very good salary because I have worked through the trivia of understanding Python and JavaScript and I'm better at that trivia than most other people.
• 0:40
  SPEAKER_0: And now you've got this machine that comes along and it's better at the trivia than I am."
• 0:44
  SPEAKER_0: I feel like there's a pessimistic and an optimistic way, the optimistic version.
• 0:48
  SPEAKER_0: I can use these tools better than anyone else for programming.
• 0:51
  SPEAKER_0: I can take my existing programming knowledge and when I combine it with these tools, I will run circles around somebody who's never written a line of code in their life.
• 0:59
  SPEAKER_0: I can just do this stuff better.
• 1:01
  SPEAKER_1: Welcome to the Pragmatic Engineering podcast.
• 1:04
  SPEAKER_1: In this show, we cover software engineering at big tech and startups.
• 1:08
  SPEAKER_1: From the inside, you'll get deep dives with experience engineers and tech professionals who share their harder and lessons, interesting stories and practical advice that they have on building software.
• 1:18
  SPEAKER_1: After each episode, you'll walk away with pragmatic approaches you can use to build stuff, whether you're a software engineer or a manager of engineers.
• 1:26
  SPEAKER_1: In this first episode, we go into a really timely topic using gen AI for coding.
• 1:31
  SPEAKER_1: Now, there's no shortage of AI companies hyping up their capabilities, but we'll size up all of that.
• 1:37
  SPEAKER_1: I turn to longtime software engineering, Simon Willison, who is safe to refer to as an independent investigator of large language models because he's been using them so much to improve his personal productivity for the last four years.
• 1:48
  SPEAKER_1: With Simon, we have a refreshingly honest conversation on how these tools actually work for us developers as of now.
• 1:56
  SPEAKER_1: We talk about common LLM use cases like fine-tuning and RAG, Simon's day-to-day large language model stack, and misconceptions about large language models.
• 2:06
  SPEAKER_1: This is the first episode of many such deep dives to come.
• 2:08
  SPEAKER_1: Subscribe to get notified of when new episodes are out.
• 2:11
  SPEAKER_1: So Simon, welcome to the podcast.
• 2:14
  SPEAKER_0: Hey, it's really great to be here.
• 2:16
  SPEAKER_1: So it's great to have you here.
• 2:18
  SPEAKER_1: You're an experienced software engineer and you've definitely been around the block.
• 2:21
  SPEAKER_1: So some people will know you from your prolific open source contributions, co-creating the Django framework, which is a rapid web development tool written in Python.
• 2:31
  SPEAKER_1: You're also the creator of a dataset tool for exploring and publishing data.
• 2:36
  SPEAKER_1: And then you're also a startup founder.
• 2:38
  SPEAKER_1: So I remember you were the founder of Lanyard, a conference direction site, which was funded by Y Combinator, acquired by Eventbrite, and then you were there for six years as an engineer, as a manager.
• 2:51
  SPEAKER_1: So you've really done all of the things, open source, founder, working at a large company.
• 2:56
  SPEAKER_0: Yeah, the startup to large company thing is particularly interesting, like moving at the speed of a startup to moving at the speed of a much larger company where bugs matter and people lose money if your software breaks.
• 3:10
  SPEAKER_1: When I started to notice you more is around the time when ChatGPT came out and you were very hands-on in trying out what this works for your development workflow, you shared a lot of things on your blog, and really this is what we're going to talk about today.
• 3:27
  SPEAKER_1: Your first hand learnings about how this AI enhanced development helps your specific workflow, where it doesn't help and what you've learned through this ... How many years has it been, two, three years of nonstop usage?
• 3:40
  SPEAKER_0: Well, so I was on GPT-3 before ChatGPT came out.
• 3:44
  SPEAKER_0: So I'm verging on three years of using this stuff frequently.
• 3:47
  SPEAKER_0: It got exciting when ChatGPT came out.
• 3:51
  SPEAKER_0: GPT-3 was interesting, but ChatGPT, that's when the whole world started paying attention to it.
• 3:57
  SPEAKER_1: To kick off, I'm interested in how you got started with these large language model tools.
• 4:05
  SPEAKER_1: What was the first time you came across them and you're like, "All right, let me give us a go"?
• 4:11
  SPEAKER_0: So I've been paying attention to the field of machine learning as a side interest for five or six years.
• 4:18
  SPEAKER_0: I did the fast AI course, Jeremy Howard's course back in I think 2018.
• 4:23
  SPEAKER_0: And then GPT-2 came out in ... Was that 2019?
• 4:29
  SPEAKER_0: Yeah, it was 2019.
• 4:31
  SPEAKER_0: GPT-2 was happening, which was the first of these models that you could see there was something interesting there, but it was not very good.
• 4:40
  SPEAKER_0: You could give a text to complete a sentence and sometimes it would be useful.
• 4:44
  SPEAKER_0: And I did an experiment back then where I tried to generate New York Times headlines for different decades by feeding in say, all of the New York Times headlines in the 1950s, then the 1960s and 1970s, and then giving it stories to complete now.
• 4:57
  SPEAKER_0: And I poked around for a bunch.
• 5:00
  SPEAKER_0: The results were not exactly super exciting and I lost interest at that point, to be honest.
• 5:06
  SPEAKER_0: And then GPT-3, which came out in 2020, but began to be more available in 2021, that's when things started getting super interesting.
• 5:16
  SPEAKER_0: Because GPT was the first of these models that was large enough that it could actually do useful things.
• 5:22
  SPEAKER_0: And one of the earliest code things I was using it for, I think I was using it for JQ, the little JSON programming language, which I've always found really difficult.
• 5:33
  SPEAKER_0: It just doesn't quite fit in my head.
• 5:34
  SPEAKER_0: And I was finding that GPT-3, if I prompted it in the right way, and this was a model where you had to do the completion prompt, so you don't ask it a question, get an answer, you say, "The JQ needed to turn this into this is ..." And then you stop and you run that in the model and it finishes the sentence, which I think is the reason most people weren't playing with it because that's a weird way of interacting with something.
• 5:55
  SPEAKER_0: In many ways, the big innovation of ChatGPT was they added a chat interface on top of this model.
• 6:02
  SPEAKER_0: And so now you didn't have to think in terms of completions, you could ask it a question and get an answer back.
• 6:08
  SPEAKER_0: But yeah, so it was very clear back then, and I was running it for about 12 months before ChatGPT came along.
• 6:15
  SPEAKER_0: There was something really interesting about this model and what it could do, and that was also the point where it became clear that code was actually something it was surprisingly good at.
• 6:24
  SPEAKER_0: And I talked to somebody at OpenAI asked them, it's like, "Were you expecting it to be good at code?"
• 6:29
  SPEAKER_0: And they said, "We thought maybe, but it wasn't one of our original goals."
• 6:33
  SPEAKER_0: The original goals of these models were much more things like translation from one human language to another, which they do incredibly well.
• 6:41
  SPEAKER_0: But when you think about it, the fact that they can write code well isn't that surprising because code is so much simpler that English or Chinese or German.
• 6:49
  SPEAKER_1: Yeah.
• 6:49
  SPEAKER_1: When we put it together, what we know, I think it is pretty obvious, and I think we will talk about implications.
• 6:56
  SPEAKER_1: But let's just jump a little bit ahead.
• 6:58
  SPEAKER_1: So I think I personally had a, "Wow, this is amazing."
• 7:02
  SPEAKER_1: Moment with LLMs, and then I've also had a bit of a scared moment of, "Could this actually replace part of what I do or not?"
• 7:13
  SPEAKER_1: And you had a really interesting story with that, a proper, "This is scary," moment.
• 7:18
  SPEAKER_1: Can you talk about that?
• 7:21
  SPEAKER_0: Definitely I've had a few of those.
• 7:22
  SPEAKER_0: I think every programmer who works with these models, the first time it spits out like 20 lines of actually good code that solves your problem and does it faster than you would, there's that moment when you're like, "Hang on a second, what am I even for?"
• 7:34
  SPEAKER_0: But I had a bigger version of that with actually with my main open source project.
• 7:40
  SPEAKER_0: So I build this tool called Dataset, which it's an interface for querying databases and analyzing data, creating JSON APIs on top of data, all of that kind of stuff.
• 7:50
  SPEAKER_0: And the thing I've always been trying to solve with that is I feel like every human being should be able to ask questions of databases.
• 7:57
  SPEAKER_0: It's absurd that everyone's got all of this data about them, but we don't give them tools that let them actually dig in and explore it and filter it and try and answer questions that way.
• 8:07
  SPEAKER_0: And then I tried this new feature of ChatGPT that they launched last year called Code Interpreter Mode.
• 8:12
  SPEAKER_0: This is the thing where ChatGPT, you can ask it a question, it could write some Python code and then it can execute that Python code for you and use the result to continue answering your question.
• 8:23
  SPEAKER_0: And Code Interpreter Mode has a feature where you can upload files to it.
• 8:27
  SPEAKER_0: So I uploaded a SQLite database file to it, just the same database files that I use in my own software.
• 8:33
  SPEAKER_0: And I asked the question and it flawlessly answered it by composing the right SQL query, running that using the Python SQLite library and spitting out the answer.
• 8:42
  SPEAKER_0: And I sat there looking this thinking, "On the one hand, this is the most incredible example of being able to ask questions of your data that I've ever seen.
• 8:50
  SPEAKER_0: But on the other hand, what am I even for?
• 8:52
  SPEAKER_0: I thought my life's purpose was to solve this problem.
• 8:55
  SPEAKER_0: And this new tool is solving my problem without even really thinking about it."
• 9:00
  SPEAKER_0: They didn't mention, "Oh, it could do SQLite SQL queries as part of what it does."
• 9:03
  SPEAKER_0: Just like Python.
• 9:05
  SPEAKER_0: And that was fun.
• 9:07
  SPEAKER_0: Well, no, that was a little bit existential dread.
• 9:10
  SPEAKER_0: And the way I've been coping with that is thinking, "Okay, well, my software needs to be better than ChatGPT code interpreted this particular problem if I mix AI features into it."
• 9:20
  SPEAKER_0: So I started exploring what plugins for my software look like that add large language model based, like run a SQL query against this schema, all of that kind of stuff.
• 9:29
  SPEAKER_0: But it's interesting.
• 9:30
  SPEAKER_0: It did very much change my mental model of the problem that I was trying to solve because it took such a big bite out of that problem.
• 9:38
  SPEAKER_1: This episode is brought to you by Codeium, the AI tool of choice or professional software developers.
• 9:43
  SPEAKER_1: That is C-O-D-E-I-U-M.
• 9:47
  SPEAKER_1: Codeium removes tedium from your development through a suite of state-of-the-art AI capabilities available via extensions in all of your favorite IDs, such as VS Code, JetBrains, Visual Studio, Eclipse, Xcode, NeoVim, Jupyter Notebooks, and more.
• 10:01
  SPEAKER_1: Uniquely, Codeium is fully enterprise ready.
• 10:04
  SPEAKER_1: As proof, it had multiple regulated Fortune 500 companies counted within its thousands of enterprise customers.
• 10:10
  SPEAKER_1: Join the 700,000 developers using Codeium's individual free plan and ask your companies to consider a free trial of the enterprise plan.
• 10:18
  SPEAKER_1: To learn more about Codeium, visit Codeium.com/pragmatic.
• 10:22
  SPEAKER_1: That is C-O-D-E-I-U-M.com/pragmatic.
• 10:28
  SPEAKER_1: And what I noticed is you have been experimenting a lot with trying out how different LLMs will work.
• 10:34
  SPEAKER_1: You've been running models locally, you've been obviously trying a lot of ... There's the usual suspect tools, but even beyond that.
• 10:40
  SPEAKER_1: Can you share a little bit on how your initial impressions were?
• 10:45
  SPEAKER_1: Because you were already on the early versions of the tool from chat GPT to Copilot to some other things, and how your stack has changed or refined to actually make you more productive, because it sounds like you are more productive now.
• 10:59
  SPEAKER_0: Yes, very much so.
• 11:01
  SPEAKER_0: So I've been calling myself an independent researcher when it comes to this kind of stuff.
• 11:06
  SPEAKER_0: Because got the time, I can dig into these things.
• 11:09
  SPEAKER_0: I write a lot.
• 11:10
  SPEAKER_0: I've been blogging about this since when I first started investigating it.
• 11:15
  SPEAKER_0: And yeah, like I said, GPT-3, I was basically using it through their playground interface, which still exists today.
• 11:21
  SPEAKER_0: It's the API debugging tool for this stuff.
• 11:25
  SPEAKER_0: And it was fine.
• 11:27
  SPEAKER_0: I experimented with having it write documentation, but I've always felt a bit funny about publishing words that I didn't write because I do so much writing myself and little bits and pieces of code.
• 11:39
  SPEAKER_0: But I didn't really get into the coding side until after ChatGPT came out and I did the Advent of Code that December and the month-long programming challenge.
• 11:50
  SPEAKER_1: So this was 2022 December, right?
• 11:52
  SPEAKER_0: Yes, November 30th is when ChatGPT came out.
• 11:55
  SPEAKER_0: And so I spent December trying to learn Rust with its assistance.
• 12:02
  SPEAKER_0: It was interesting.
• 12:03
  SPEAKER_0: I got a reasonably long way.
• 12:06
  SPEAKER_0: I still don't know Rust.
• 12:07
  SPEAKER_0: The memory management in Rust is just difficult enough that language models still have trouble with it.
• 12:14
  SPEAKER_0: One of my tests of a new language model is, "Okay, can it explain the Rust borrowing to me?"
• 12:19
  SPEAKER_0: And they're getting to a point where I'm almost understanding it, but it's an interesting stress test for this.
• 12:25
  SPEAKER_0: Whereas if you use these models for JavaScript and Python, they're phenomenally good.
• 12:29
  SPEAKER_0: There's so much more training data about JavaScript and Python out there than there is for a language like Rust that honestly they just completely sing.
• 12:37
  SPEAKER_0: And that's great for me because the languages I use every day are Python and JavaScript and SQL, and those are the three languages that language models are best at.
• 12:48
  SPEAKER_0: So I'm perfectly positioned to have these things be useful and helpful for me.
• 12:53
  SPEAKER_0: And I tend to pick, like I said, boring technology like Django, which the language moms know already.
• 13:00
  SPEAKER_0: If you stick with Django, they're going to be able to do pretty much anything that you ask of them.
• 13:06
  SPEAKER_0: But yeah, so I tried learning Rust and that was a really good exercise for just every day trying these things out and seeing what could happen.
• 13:13
  SPEAKER_0: One of the key things I've learned that I think people don't necessarily acknowledge, these things are really difficult to use and there's a lot of ... It's not just skill.
• 13:23
  SPEAKER_0: There's a lot of intuition you have to build up in order to use them effectively.
• 13:27
  SPEAKER_0: If you just sit down and ask it a question like you'd ask on Stack Overflow, you'll probably not get a great response.
• 13:33
  SPEAKER_0: And a lot of people do that, and then they write the whole thing off.
• 13:36
  SPEAKER_0: They're like, "Okay, it didn't give me what I wanted.
• 13:39
  SPEAKER_0: This is all hype.
• 13:39
  SPEAKER_0: There's no value here."
• 13:41
  SPEAKER_0: The trick is, firstly, you have to learn how to prompt them.
• 13:44
  SPEAKER_0: More importantly, you have to learn what kind of things they're good at and what kind of things they're bad at.
• 13:49
  SPEAKER_0: I know because I've spent so much time with them, that Python, JavaScript, they're great at, Rust, they're not quite as good at yet.
• 13:55
  SPEAKER_0: I know that you shouldn't ask them about current events because they've got a training cut off in terms of what they understand.
• 14:01
  SPEAKER_0: I know that they're terrible at math and logic puzzles.
• 14:05
  SPEAKER_0: Don't ask them to count anything, which is bizarre because computers are really good at maths and counting and looking things up.
• 14:13
  SPEAKER_0: And language models, those are the three things they're not good at, and they're supposed to be our most advanced computers.
• 14:18
  SPEAKER_0: But so you have to build this quite intricate mental model of what these things can do and how to get them to do those things.
• 14:25
  SPEAKER_0: And if you build that mental model, if you put the work in, you can scream at them.
• 14:30
  SPEAKER_0: You can work so quickly at solving specific problems.
• 14:34
  SPEAKER_0: When you say, "Oh, this is the kind of thing that a language model can do."
• 14:37
  SPEAKER_0: And then you just outsource it to your, I call it my weird intern sometimes.
• 14:41
  SPEAKER_0: Whereas other things, you're like, "Okay, well, it's not even worth trying out on the language model because I know from past experience that it won't do a good job with it."
• 14:48
  SPEAKER_1: As a software engineer, we do have a bit of an enduring mindset, but when you see a new technology, clearly this is here, it's not going away.
• 14:57
  SPEAKER_1: But there's two ways you can look at it.
• 14:59
  SPEAKER_1: One, I think you explained.
• 15:01
  SPEAKER_1: You start playing with it, you start stress testing it, you see where it works, where it doesn't.
• 15:05
  SPEAKER_1: And the other one is you start from a theory, you understand how it's built, how it works, what's behind the scenes, and then you start probing and then you have ... I think this is a little bit with the way computer science is taught.
• 15:18
  SPEAKER_1: If you go to university, when I went to computer science, we started with algebra and some formal methods and languages and coding was a little bit, we got there by the end and you're like, "Well, yeah, I guess I now know what happens underneath the compiler."
• 15:34
  SPEAKER_1: But obviously there's the other route as well.
• 15:37
  SPEAKER_1: In your view, it sounds like you jumped straight into, "Let me see how this actually works, and let me not overthink the theory."
• 15:47
  SPEAKER_1: Which at the time it was a bit unclear.
• 15:52
  SPEAKER_0: Right, and actually right now, if you start with the theory, it will hold you back.
• 15:55
  SPEAKER_0: This specific technology, it's weirdly harmful to spend too much time trying to understand how they actually work before you start playing with them, which is very unintuitive.
• 16:08
  SPEAKER_0: I have friends who say that if you're a machine learning researcher, if you've been training models and stuff for years, you're actually disadvantaged to start using these tools than if you come in completely fresh, because they're very weird.
• 16:22
  SPEAKER_0: They don't react like you expect other machine learning models ... Machine learning people always jump straight to fine-tuning.
• 16:29
  SPEAKER_0: Fine-tuning on these things is mostly a waste of time.
• 16:32
  SPEAKER_0: It takes people a long time to get to the point where like, "You know what?
• 16:35
  SPEAKER_0: There's no point in fine-tuning my own custom version of this because next month-
• 16:41
  SPEAKER_1: Just to break it over fine-tuning, because I think we hear this word a lot, but by fine-tuning you mean that you take the model and then you add more training to ... You run training cycles?
• 16:54
  SPEAKER_0: Exactly.
• 16:54
  SPEAKER_0: And it's a very confusing term because ... Yeah, so the idea with fine-tuning is you take an existing model, it might be one of the openly licensed models, or actually I think Claude has this now and OpenAI have APIs where you can upload a CSV file of a million examples and spend a lot of money with them, and they will give you a model fine-tuned on that, and it sounds so tempting.
• 17:16
  SPEAKER_0: Everyone's like, "Wow, I could have a model that's perfectly attuned to my specific needs."
• 17:21
  SPEAKER_0: It's really difficult to do.
• 17:22
  SPEAKER_0: It's really expensive.
• 17:24
  SPEAKER_0: And for most of the things that people want to do, it turns out it doesn't actually solve the problem.
• 17:28
  SPEAKER_0: Lots of people think, " I want the model to know about my documentation or my company's internal documentation.
• 17:34
  SPEAKER_0: I want to ask questions about that.
• 17:36
  SPEAKER_0: Surely, I fine tune a model to solve that."
• 17:39
  SPEAKER_0: That, it turns out, just plain doesn't work because the weight of all of the existing knowledge the model has completely overwhelms anything that you try and add into it with fine-tuning.
• 17:49
  SPEAKER_0: The models, they hallucinate more on questions about things if you've done that extra fine-tuning step to add knowledge, which is a surprising thing.
• 17:57
  SPEAKER_0: Where fine-tuning does work is for tasks.
• 18:01
  SPEAKER_0: If you want a model that's just really good at SQL, you can give it 10,000 examples of, "Here's a human question of the SQL schema and here's the SQL query."
• 18:09
  SPEAKER_0: And that will give you a model that is stronger at that kind of activity, but for adding new facts into the model, it just doesn't work, which confuses people.
• 18:18
  SPEAKER_0: And so then you have to look at the other techniques for solving that problem.
• 18:22
  SPEAKER_0: There's a thing called RAG, which is a very fancy acronym for a very simple trick.
• 18:27
  SPEAKER_0: It stands for retrieval augmented generation.
• 18:30
  SPEAKER_0: All it means is the user asks a question, you search your documentation for things that might be relevant to that question.
• 18:36
  SPEAKER_0: You copy and paste the whole lot into the model, and these models can take quite a lot of input now, and then put the user's question at the end.
• 18:42
  SPEAKER_0: That's it.
• 18:43
  SPEAKER_0: Super, super simple.
• 18:44
  SPEAKER_0: Don't get baffled-
• 18:46
  SPEAKER_1: It's so simple-
• 18:47
  SPEAKER_0: It's so simple.
• 18:47
  SPEAKER_1: ... I actually wrote an article about it and I had one of the people who guest wrote it built an open source tool, it was just a tool to do your own RAG training and you could plug in ChatGPT and I did it.
• 18:59
  SPEAKER_1: I understand the code, and the code itself was very simple and I was like, "Is this-
• 19:03
  SPEAKER_1: The code, and the code itself was very simple.
• 19:03
  SPEAKER_0: Why?
• 19:03
  SPEAKER_1: And I was like, is this all there is to it?
• 19:04
  SPEAKER_1: You just break it up into chunks, you get some embeddings so you can figure out where search will end you, and then you just add in that extra thing.
• 19:12
  SPEAKER_1: And the only thing, obviously you can go down the rabbit hole, but for Simple Rag is you decide on the context window size, for the most part.
• 19:20
  SPEAKER_0: Yeah, I mean-
• 19:21
  SPEAKER_1: And I was amazed at how well, as you said, it seemed so simple.
• 19:24
  SPEAKER_1: So I looked at the code and I said, "Well this ... I'm not expecting much."
• 19:27
  SPEAKER_1: And when I tried it out, it worked really well.
• 19:30
  SPEAKER_1: It's one of those counter ... It feels there's some counter intuitive things.
• 19:35
  SPEAKER_0: Yeah, so Rag, it's the Hello World of building software on top of LLMs.
• 19:39
  SPEAKER_0: You don't get it to print Hello World, you get it to answer questions about your documentation.
• 19:43
  SPEAKER_0: I've implemented like 30 lines of Python.
• 19:45
  SPEAKER_0: I've got one version that's like two dozen lines of Bash, I think.
• 19:48
  SPEAKER_0: It's very easy to get the basic version working, but getting good Rag working is incredibly difficult because the problem is that if you built the system and you know how it works, you are naturally going to ask questions of it in the right kind of format.
• 20:01
  SPEAKER_0: The moment you expose it to real human beings, they will come up with an infinite quantity of weird ways that they might ask questions.
• 20:09
  SPEAKER_0: And so the art of building good Rag systems, the reason that it could take six months to actually get it production ready is figuring out, "Okay, there were all of these different ways that it can go wrong."
• 20:19
  SPEAKER_0: And the key trick in RAG is always how do we fill that context?
• 20:23
  SPEAKER_0: How do we pick the information that's most relevant to what the user is asking, which is really hard.
• 20:28
  SPEAKER_0: That's actually like ... It's an information retrieval problem.
• 20:30
  SPEAKER_0: It's what search engineers have been trying to figure out for 30 years, and there's a lot of depth to that field.
• 20:36
  SPEAKER_0: So Rag, just like everything else in language models, it's fractally interesting and complicated.
• 20:43
  SPEAKER_0: It's simple at the top, and then each little aspect of it gets more and more involved the further you look.
• 20:48
  SPEAKER_0: One of my favorite difficult problems in this is what's called in the industry, evals.
• 20:53
  SPEAKER_0: Automated evaluations.
• 20:54
  SPEAKER_0: Because when you're writing software, we write automated tests, we write unit tests and they tell us if our software works and that's great.
• 21:00
  SPEAKER_0: You can't do that with language models, because they're non-deterministic.
• 21:06
  SPEAKER_0: They very rarely return exactly the same answer.
• 21:08
  SPEAKER_0: So we don't even have unit testing.
• 21:10
  SPEAKER_0: But with things like rag, we need to have automated tests that can tell us, "Okay, we tweaked our algorithm for picking content."
• 21:18
  SPEAKER_0: Is it better?
• 21:19
  SPEAKER_0: Does that do a better job of answering questions?
• 21:21
  SPEAKER_0: It's really difficult.
• 21:22
  SPEAKER_0: I'm still trying to figure out the right patterns of this myself.
• 21:26
  SPEAKER_1: And I talked with someone who's working out at AI company.
• 21:28
  SPEAKER_1: And the weird thing that it just feels it breaks, all that we know is they have this eval test suite, which runs against their model and whenever they make a change, they run it and told me like, "Okay, it costs us $50 to run this every single time."
• 21:43
  SPEAKER_0: Wow.
• 21:44
  SPEAKER_1: Yeah.
• 21:44
  SPEAKER_1: This is just something I don't think we've been used to.
• 21:47
  SPEAKER_1: I run my test as a software engineer, I'm a unit test integrator.
• 21:50
  SPEAKER_1: I know how much time it costs me, but suddenly, obviously they're using different API's, whichever vendor.
• 21:57
  SPEAKER_1: It feels like there's a bit of a ... This clearly used to be the thing before my time, at least, back when there were servers or mainframes or computing time was expensive, but suddenly this is just yet another interesting vibe also.
• 22:11
  SPEAKER_0: Yep.
• 22:11
  SPEAKER_0: Yeah.
• 22:12
  SPEAKER_0: So you don't want to run those on every commit to your repository that will bankrupt you pretty quickly.
• 22:16
  SPEAKER_1: Yeah, yeah.
• 22:17
  SPEAKER_0: It's also funny that with evals, one of the most common techniques is what's called LLM as a judge.
• 22:22
  SPEAKER_0: So if you're trying to say, "Okay, I'm building a summarizer.
• 22:25
  SPEAKER_0: Here's an article, I want it summarized, here's the summary," how could you write tests against a summary to check that it's actually good?
• 22:33
  SPEAKER_0: And what a lot of people do is they outsource that to another model.
• 22:36
  SPEAKER_0: So they produce two summaries, and then they say, "Hey GPT-4, which of these two summaries is best?"
• 22:42
  SPEAKER_0: And I find that so uncomfortable.
• 22:44
  SPEAKER_0: This stuff is all so weird and difficult to evaluate already, and now we're throwing in another letter of weird language models to try and give us a score from our previous language models.
• 22:53
  SPEAKER_0: But these are the kind of options that we're exploring at the moment.
• 22:57
  SPEAKER_1: Yeah, it's interesting.
• 22:58
  SPEAKER_1: Speaking about options, so you've experimented a lot with trying out different tools, including building your own, and obviously copilot and other models.
• 23:08
  SPEAKER_1: I saw you mentioned Claude, for example, as one you're playing with.
• 23:12
  SPEAKER_1: What is your current LLM stack, and day to day how do you use it for actually coding on data set or on your projects?
• 23:21
  SPEAKER_0: So my default stack right now is ... My default model is Claude 3.5 Sonnet, which is brand new.
• 23:28
  SPEAKER_0: It came out maybe three weeks ago.
• 23:30
  SPEAKER_1: I heard it's amazing for coding.
• 23:32
  SPEAKER_0: It's amazing for everything.
• 23:33
  SPEAKER_0: It is the first time somebody who is not Open AI has had the clearly best model.
• 23:39
  SPEAKER_0: It's just better than Open AI's best available models at the moment.
• 23:42
  SPEAKER_0: The team behind it, the company behind it, Anthropic, are actually a splinter group from Open AI.
• 23:47
  SPEAKER_0: They split a couple of years ago and apparently it's because they tried to get Sam Altman fired, which you can't do.
• 23:54
  SPEAKER_0: We saw this happen publicly.
• 23:55
  SPEAKER_1: We saw that.
• 23:56
  SPEAKER_0: Six months ago.
• 23:57
  SPEAKER_0: But they were early adopters.
• 23:59
  SPEAKER_0: Two and a half years ago.
• 24:00
  SPEAKER_0: They tried to get Sam Altman fired.
• 24:01
  SPEAKER_0: It didn't work, they quit and split up their own company.
• 24:04
  SPEAKER_0: And they were some of the people who built the built GPT-4, so it's actually the GPT-4 original team.
• 24:11
  SPEAKER_0: But anyway, Claude 3.5 Sonnet is unbelievably good.
• 24:15
  SPEAKER_0: It's my default for most of the work that I'm doing.
• 24:18
  SPEAKER_0: I still use GPT 4.0, which is open AI's ... Probably their best available model, mainly for two features.
• 24:27
  SPEAKER_0: It's got code interpret mode, this thing where it can write Python and code and then execute that Python code.
• 24:32
  SPEAKER_0: So sometimes I'll throw a fiddly problem at it and I'll watch it, try five or six times until it works, and I just sit there and watch it going through the motions.
• 24:41
  SPEAKER_0: So I use that a lot.
• 24:42
  SPEAKER_0: And then Chat GPT has the voice mode, which I use when I'm walking my dog.
• 24:49
  SPEAKER_0: Because you can stick in a pair of AirPods and you can go for an hour long walk with the dog and you can talk to this weird AI assistant and have it write you code, because it can do code interpreter, and it can look things up on the internet and such.
• 25:01
  SPEAKER_0: So you can have a very productive hour-long conversation while you're walking the dog on the beach.
• 25:06
  SPEAKER_1: This I was not expecting.
• 25:07
  SPEAKER_1: I'll be honest.
• 25:08
  SPEAKER_0: Yeah, that is the most dystopian sci-fi future thing as well.
• 25:14
  SPEAKER_0: The voice mode.
• 25:15
  SPEAKER_0: And this isn't the fancy new voice mode they demoed a few weeks ago.
• 25:18
  SPEAKER_0: This is the one they've had for six months.
• 25:20
  SPEAKER_0: It's so good.
• 25:22
  SPEAKER_0: The intonation, the voice.
• 25:25
  SPEAKER_1: Yeah, wow.
• 25:25
  SPEAKER_0: It's like having a conversation with an intern who can go and look things up for you.
• 25:30
  SPEAKER_1: So you mentioned the stack, but if I imagine your data, you've got your terminal or your code-
• 25:36
  SPEAKER_0: There's more to my stack.
• 25:37
  SPEAKER_0: So those are the ones I'm using in my browser and on my phone.
• 25:41
  SPEAKER_1: Got it.
• 25:42
  SPEAKER_0: I use GitHub copilot.
• 25:44
  SPEAKER_0: I've always got that turned on.
• 25:46
  SPEAKER_0: I use my ... I've built been building this open source tool called LLM, which is command line [inaudible 00:25:51]-
• 25:50
  SPEAKER_1: Just a question.
• 25:51
  SPEAKER_1: On copilot, what features do you use?
• 25:53
  SPEAKER_1: Because it now has a competing feature, it does have a chat window if you want to use that.
• 25:56
  SPEAKER_1: It has auto-complete.
• 25:57
  SPEAKER_1: Which ones do you find most useful for your use cases?
• 26:00
  SPEAKER_0: Mostly auto-complete, like old school copilot.
• 26:03
  SPEAKER_0: I've recently started using the thing where you can select some lines of code and there's a little sparkly icon.
• 26:08
  SPEAKER_0: You can click that and then give it a prompt to run against those lines of code and it'll do that.
• 26:12
  SPEAKER_0: I don't use the chat window at all.
• 26:13
  SPEAKER_0: I use Claude in the browser for what I would use that for and it's great.
• 26:22
  SPEAKER_0: Copilot's another interesting one where you hear from people who are like, "I turned it on and it just gave me a bunch of junk and I turned it off again because it's clearly not useful."
• 26:29
  SPEAKER_0: And again, Copilot, you have to learn how to use it.
• 26:32
  SPEAKER_0: There's no manual for any of this stuff, but especially not for Copilot.
• 26:35
  SPEAKER_0: And you have to learn things like if you type out the start of a function name and give it carefully named parameters with their types, or type annotations, it will complete the function for you.
• 26:47
  SPEAKER_0: And if you add a comment ... You prompt it through the comments that you write.
• 26:52
  SPEAKER_0: Essentially.
• 26:53
  SPEAKER_1: Yeah, I've actually started to use that.
• 26:55
  SPEAKER_1: It's actually, again, no one tells you that, but once you figure it out it can be really useful how you can generate either a small, for me, just a small part or a function.
• 27:05
  SPEAKER_1: It just gets it, and again, it's not surprising, but the more context you give in the comment, the more it'll kind of do what you want.
• 27:12
  SPEAKER_1: If you're lucky.
• 27:13
  SPEAKER_0: I think the other thing to know about Copilot is that it's actually running Rag.
• 27:17
  SPEAKER_0: It's got an incredibly sophisticated retrieval Rag mechanism where every time it does a completion for you it tries to include context from nearby in your file, but it also looks for other files in your project that have similar keywords in them.
• 27:35
  SPEAKER_0: So that's why sometimes you'll be in your test-
• 27:37
  SPEAKER_1: That's really interesting that you say that, because we're going to get to the misconceptions, but we've been running an AI survey and one of the things that people really complain about saying is, "I use Copilot because it's the one that's easiest to turn on in your ID," and people said that, "It only uses my files and I wish it would look at the project or understand the whole project."
• 27:56
  SPEAKER_1: But it's interesting you say that, because I think a lot of people don't realize that it is trying to do it in smart ways.
• 28:02
  SPEAKER_1: Most people, or not most, but a lot of people assume that it just only looks at whatever you're seeing on the screen.
• 28:07
  SPEAKER_0: No, it is looking at bits of other files, but it's undocumented and it's weird and it's trying to do semantic similarities and all of that sort of stuff.
• 28:14
  SPEAKER_0: What I do a lot of is sometimes I'll just copy and paste a chunk of one file into a comment in another so that it's definitely visible to copilot.
• 28:21
  SPEAKER_0: That's great.
• 28:22
  SPEAKER_0: For things like writing tests.
• 28:23
  SPEAKER_0: You can literally copy in the code that you're testing into your tests.py and then start writing [inaudible 00:28:28] and it'll fill in the details.
• 28:29
  SPEAKER_1: Yeah.
• 28:29
  SPEAKER_1: So I'm now starting to understand.
• 28:30
  SPEAKER_1: When you said you need to learn how to use it, sounds like you are coming from the other way instead of trying out and saying yay or nay, because I guess you're working for yourself, so it kind of makes sense that you want to make yourself productive.
• 28:41
  SPEAKER_1: You figure it out how these things can actually make you more productive,
• 28:47
  SPEAKER_0: Right.
• 28:47
  SPEAKER_0: Absolutely.
• 28:48
  SPEAKER_0: And it's so much work.
• 28:49
  SPEAKER_0: I think that the biggest sort of misconception about all of this is that you'll get this tool and it'll make you productive on day one, and it absolutely won't.
• 28:57
  SPEAKER_0: You have to put in so much effort to learn to explore it and experiment and learn how to use it, and there's no guidance.
• 29:04
  SPEAKER_0: Like I said, Copilot doesn't have a manual, which is crazy.
• 29:07
  SPEAKER_0: Claude, to its credit, Claude is the only one of these things that actually has documentation that's really good.
• 29:13
  SPEAKER_0: If you want to learn how to prompt LLMs, the Claude Anthropic Prompting Guide is actually the best thing I've seen anywhere.
• 29:19
  SPEAKER_0: OpenAI have almost nothing.
• 29:21
  SPEAKER_0: There are so many hypesters and blogs and tweets and LinkedIn posts full of junk advice.
• 29:28
  SPEAKER_0: All of the things like, "Always tell it that you are the world's greatest expert in X before you ask it."
• 29:33
  SPEAKER_0: All of that kind of thing.
• 29:34
  SPEAKER_0: Mostly rubbish, right?
• 29:36
  SPEAKER_1: Yeah.
• 29:37
  SPEAKER_0: But there's so much superstition.
• 29:38
  SPEAKER_0: Because this stuff isn't documented and even the people who created the models don't fully understand how they do what they do.
• 29:45
  SPEAKER_0: It's very easy to form superstitions.
• 29:47
  SPEAKER_0: You try the, "You are the world's greatest expert in Python thing," and you get a good answer, so you're like, "Okay, I'll do that from now on."
• 29:53
  SPEAKER_0: It's kind of like if your dog finds a hamburger in a bush, every time you walk past that bush for the next two years, they will check for a hamburger, right?
• 30:02
  SPEAKER_0: Because dogs are very superstitious.
• 30:04
  SPEAKER_0: And it's that, but for software engineering.
• 30:08
  SPEAKER_1: And then going back to your stack.
• 30:09
  SPEAKER_1: So-
• 30:10
  SPEAKER_0: Yeah. Couple more tools.
• 30:10
  SPEAKER_1: You know these tools ...
• 30:17
  SPEAKER_0: There's a few more.
• 30:17
  SPEAKER_0: So I talked about Co-interpreter.
• 30:18
  SPEAKER_0: One of my favorite Claude features is, again, a new feature from a few weeks ago called Artifacts, which is this thing where Claude can now write HTML and CSS in JavaScript, and then it can show you that in a little secure iFrame.
• 30:30
  SPEAKER_0: And so it can build you tools in one of interfaces and prototypes on demand.
• 30:35
  SPEAKER_0: And it's quite limited.
• 30:37
  SPEAKER_0: They can't make API calls from in there.
• 30:39
  SPEAKER_0: It can't actually see the results, so it doesn't have that debug loop that code interpreter has.
• 30:43
  SPEAKER_0: But still, it's amazing.
• 30:45
  SPEAKER_0: I've redesigned pages on my blog by pasting in a screenshot of my blog and then saying, "Suggest better color scheme for this and show me a prototype in an artifact," and it did.
• 30:57
  SPEAKER_0: So cool.
• 30:58
  SPEAKER_0: So I'm doing a lot more front-end stuff now because I can get Claude to build me little interactive prototypes along the way to help speed that up.
• 31:07
  SPEAKER_0: So I'm spending a lot of time with that.
• 31:08
  SPEAKER_0: I have my command line tool.
• 31:09
  SPEAKER_0: LLM lets you run prompts from the command line, and the key feature of that is that you can pipe things into it.
• 31:15
  SPEAKER_0: So I can cat a file into that and say, "LLM, write the tests," and it will output the tests for that file.
• 31:23
  SPEAKER_1: And then just understand, you just build like it's a command line.
• 31:25
  SPEAKER_1: Are you running a local model, or a model on a server?
• 31:29
  SPEAKER_0: LLM, the tool ... It's based around plugins, and it can talk to over a hundred different models.
• 31:34
  SPEAKER_1: So is this an open source tool?
• 31:35
  SPEAKER_0: Yes, it's open my open source language model, command line project.
• 31:40
  SPEAKER_1: Oh, awesome.
• 31:40
  SPEAKER_1: All right.
• 31:40
  SPEAKER_1: We'll link it in the show notes as well.
• 31:42
  SPEAKER_0: And yeah, so it's plugin based.
• 31:43
  SPEAKER_0: Originally it could just do Open AI, and then I added plugins, and now it can run local models and it can talk to other models too.
• 31:50
  SPEAKER_0: So I mainly use it with Claude because that's the best available model.
• 31:54
  SPEAKER_0: But I've also run like Microsoft's Phi-3, and Llama, and Mistral and things.
• 31:59
  SPEAKER_0: I can run those locally, which to be honest, I don't use on a day-to-day basis because they're just not as good.
• 32:05
  SPEAKER_0: The local models are very impressive, but the really high-end, the best of the best models run circles around them.
• 32:13
  SPEAKER_0: So when I'm trying to be productive, I'm mostly working with the best available models.
• 32:18
  SPEAKER_0: I love running the local models for research and for playing around.
• 32:22
  SPEAKER_0: And also they're a great way to learn more about how language models actually work and what they can do.
• 32:27
  SPEAKER_0: Because when you ... And people talk about hallucination a lot.
• 32:31
  SPEAKER_0: I think it's really useful to have a model hallucinate at you early, because it helps you get that better mental model of what it can do.
• 32:39
  SPEAKER_0: And the local models hallucinate wildly.
• 32:41
  SPEAKER_0: So if you really want to learn more about language models, running a tiny little ... Some of them are like two or three gigabyte files that you can run on a laptop.
• 32:48
  SPEAKER_0: I've got one that runs on my phone.
• 32:50
  SPEAKER_0: It's actually-
• 32:51
  SPEAKER_1: Really?
• 32:51
  SPEAKER_1: Which one?
• 32:52
  SPEAKER_0: Yeah.
• 32:53
  SPEAKER_0: There's an app called MLC.
• 32:56
  SPEAKER_0: MLC Chat.
• 32:57
  SPEAKER_0: And it can run Microsoft 5.3 And Google's Gemma, and it's got Mistral 7B.
• 33:03
  SPEAKER_0: These are very good models.
• 33:07
  SPEAKER_0: If you say, "Who is Simon Wilson," they will make up things.
• 33:10
  SPEAKER_0: I use ego searches to basically see how much they hallucinate.
• 33:15
  SPEAKER_0: They'll say, "He was the CTO of GitHub," and I'm like, "Well, I really wasn't," but I do use GitHub.
• 33:22
  SPEAKER_0: But I've used these on planes.
• 33:24
  SPEAKER_0: They're good enough at Python that I can use them to look up little bits of API documentation that they can't remember and things like that.
• 33:30
  SPEAKER_0: And it runs on your phone.
• 33:31
  SPEAKER_0: It's really fun.
• 33:33
  SPEAKER_1: Yeah.
• 33:33
  SPEAKER_1: Awesome.
• 33:34
  SPEAKER_1: So looking back, you've now been coding for more than 20 years, depending on how you count but-
• 33:42
  SPEAKER_0: Professionally.
• 33:42
  SPEAKER_0: People have been paying me for 20 years at this point.
• 33:44
  SPEAKER_1: People have been paying for 20 years.
• 33:44
  SPEAKER_1: So throughout this time, we have seen some increases in productivity, may that be Firebug coming out for developers, or other things.
• 33:55
  SPEAKER_1: Could you talk through what were bumps when you became more productive as a developer?
• 34:00
  SPEAKER_1: And then when we get to LLMs, compare to how this bump compares to those ones.
• 34:06
  SPEAKER_0: I love that you mentioned Firebug, because that was a big bump.
• 34:09
  SPEAKER_0: Yeah, Firebug was the Chrome DevTools before browsers had them built in.
• 34:15
  SPEAKER_0: It was an extension for Firefox that added essentially what you recognize as the developer rules now, and that was an absolute revelation when it came out.
• 34:23
  SPEAKER_0: Especially for me because I've spent most of my career as a Python program.
• 34:27
  SPEAKER_0: My favorite feature of Python is the interactive prompt.
• 34:30
  SPEAKER_0: I love being able to code by writing a line of code and hitting enter and seeing what it does, and then you end up copying and pasting a bunch of those explorations into a file, but you know that it's going to work because you worked on it interactively.
• 34:42
  SPEAKER_0: Firebug instantly brought that to JavaScript.
• 34:44
  SPEAKER_0: Suddenly you could interactively code against a live web page and figure things out that way.
• 34:48
  SPEAKER_0: So that was a big one.
• 34:50
  SPEAKER_0: I think the biggest product-
• 34:53
  SPEAKER_1: Yeah, I think as a reminder, because some listeners were not necessarily around, but before Firebug I was doing web development and the way you debugged your JavaScript applications, which were pretty simple at the time, but you did alerts to show-
• 35:05
  SPEAKER_0: Right.
• 35:05
  SPEAKER_0: We didn't even have console.log.
• 35:07
  SPEAKER_0: Console.log was invented by Firebug.
• 35:10
  SPEAKER_1: Yeah.
• 35:11
  SPEAKER_1: So it was just really painful and really hard to debug, and you also couldn't really inspect the elements, so you were changing ... It was like doing it in the dark.
• 35:18
  SPEAKER_1: And as you say, it was a game changer.
• 35:20
  SPEAKER_1: And now these days, Chrome Developer tools is better than what Firebug used to be, but Firebug was almost as good as the Chrome Developer tools today, in my memory at least.
• 35:29
  SPEAKER_1: So it was this huge leap.
• 35:31
  SPEAKER_1: And I think for front-end developers, it's hard to tell how much more, but I'm sure at least twice the productivity, I'll just say.
• 35:38
  SPEAKER_1: Because it took so much longer to fix things or to understand why things were happening.
• 35:43
  SPEAKER_1: So yeah, that was a big jump.
• 35:44
  SPEAKER_0: So Firebug's a good one.
• 35:45
  SPEAKER_0: The biggest productivity boost in my entire career is just open source generally.
• 35:49
  SPEAKER_0: So it turns out, 25 years ago you had to really fight to use anything open source at all.
• 35:56
  SPEAKER_0: A lot of companies had blanket bans on open source anything.
• 36:00
  SPEAKER_0: Microsoft were making the case that this is a very risky thing for you to even try.
• 36:08
  SPEAKER_0: That's completely gone out of the window.
• 36:09
  SPEAKER_0: I don't think there's a company left on earth that can have that policy now, because how are you going to run any front-end code without NPM?
• 36:16
  SPEAKER_0: That's all.
• 36:18
  SPEAKER_0: But that ... So it was open source as a concept, and I was very early on in open source, Django was a ... Django open source in 2005, Python and PHP and so forth all came out of the open source community.
• 36:30
  SPEAKER_0: And that was huge because prior to open source, the way you wrote software is you sat down and you implemented the same thing that everyone else had already built, or if you had the money, you bought something from a vendor.
• 36:42
  SPEAKER_0: But good luck buying a decent thing.
• 36:44
  SPEAKER_0: And then of course you can't customize it because it's proprietary.
• 36:47
  SPEAKER_0: And that, the open source ...
• 36:49
  SPEAKER_0: And then on top of open source as a concept, it really was GitHub coming along massively accelerated open source because prior to that it was SourceForge and mailing lists and CVS and subversion, just starting a new project ... I started open source projects where I had to start by installing track, which meant I needed to run a virtual private server, and then get Linux secured, and then install the open source alternative of what GitHub became.
• 37:16
  SPEAKER_0: It was great software, but it was not exactly a one-click experience.
• 37:21
  SPEAKER_0: So open source was absolutely huge.
• 37:22
  SPEAKER_0: And then you had GitHub making open source way more productive and accessible and massively accelerating it.
• 37:28
  SPEAKER_0: Then the package managers.
• 37:29
  SPEAKER_0: So PyPI for Python and NPM for JavaScript.
• 37:34
  SPEAKER_0: The OG of that was CPAN for Perl, which was up and running in the late '90s.
• 37:40
  SPEAKER_0: And we owe so much to CPAN and how it made that kind of thing happen.
• 37:46
  SPEAKER_0: But today, the productivity boost you get from just being able to PIP install or NPM install a thing that solves your problem, I think ... My hunch is that developers who grew up with that already in place have no idea how much of a difference that makes.
• 38:01
  SPEAKER_0: When I did my software engineering degree 20-
• 38:03
  SPEAKER_0: ... that makes.
• 38:03
  SPEAKER_0: When I did my software engineering degree 20 years ago, one of the big challenges everyone talked about was software reusability.
• 38:10
  SPEAKER_0: Why are we writing the same software over and over again?
• 38:13
  SPEAKER_0: And at the time, people thought OOP was the answer.
• 38:16
  SPEAKER_0: They were like, "Oh, if we do everything as classes in Java, then we can subclass those classes, and that's how we'll solve reusable software."
• 38:22
  SPEAKER_0: With hindsight, that wasn't the fix.
• 38:24
  SPEAKER_0: The fix was open source.
• 38:25
  SPEAKER_0: The fix was having a diverse and vibrant open source community releasing software that's documented and you can package and install and all of those kinds of things.
• 38:34
  SPEAKER_0: That's been incredible.
• 38:36
  SPEAKER_0: The cost of building software today is a fraction of what it was 20 years ago, purely thanks to open source.
• 38:45
  SPEAKER_1: It's interesting because when we talk about developer productivity, it's a topic that will come back and obviously it's very popular, very important for people in leadership positions who are hiring a certain number of people, and their CEOs will ask, "How are these people used?"
• 39:02
  SPEAKER_1: And right now there is a big push to say that gen AI is adding this and this much productivity.
• 39:09
  SPEAKER_1: But two things are interesting.
• 39:10
  SPEAKER_1: One is that we don't really talk about how much just having open source or not having to do it adds.
• 39:16
  SPEAKER_1: I guess we just take it for granted.
• 39:18
  SPEAKER_1: And the other thing that I want to ask you, I want to ask you how much more productive do you think with this current workflow you have, which is pretty advanced.
• 39:24
  SPEAKER_1: It sounds like it.
• 39:24
  SPEAKER_1: You're using a bunch of different tools.
• 39:26
  SPEAKER_1: You spend a lot of time tweaking it, so I'm going to assume you're one of the software engineers who are using it more efficiently to your own personal productivity.
• 39:36
  SPEAKER_1: How do you feel how much more productive this makes you?
• 39:39
  SPEAKER_1: And there's a caveat here.
• 39:41
  SPEAKER_1: Obviously, it's hard to be honest about yourself, but right now the good thing is we don't have any pulls.
• 39:49
  SPEAKER_1: Vendors will obviously have a bias to say, AI vendors, that it's helping them more.
• 39:54
  SPEAKER_1: And people who might not like these tools, they might have a bias to say like, "Ah, it's not even helping me."
• 39:59
  SPEAKER_1: So I think the best answer we can probably get right now is just from people like you looking honestly at yourself and getting a sense.
• 40:06
  SPEAKER_0: Okay, so I think I've got two answers to this.
• 40:10
  SPEAKER_0: It's difficult to quantify this, but my guess for a while has been that I've had a giant productivity boost in the portion of my job which is typing code at a computer.
• 40:22
  SPEAKER_0: And I would estimate I am two to three times more productive, faster at turning thoughts into working code, than I was before.
• 40:29
  SPEAKER_0: But that's only 10% of my job.
• 40:31
  SPEAKER_0: As a software engineer, once you're a more senior software engineer, you spend way more time researching and figuring out what the requirements for the thing are and all of those other activities.
• 40:44
  SPEAKER_0: So a huge boost for typing code.
• 40:49
  SPEAKER_0: The other thing, and it does speed up a lot of the other activities, the research activity in particular.
• 40:56
  SPEAKER_0: If I need a little JavaScript library to solve a particular problem because I have a bias towards boring technology anyway, if I ask Claude or GPT-4, it always asks for options.
• 41:08
  SPEAKER_0: I always say, "Give me options for solving this problem," and it spits out three or four, and then I can go and look at those.
• 41:14
  SPEAKER_0: And it's effectively using it as a slightly better, slightly faster and more productive Google search.
• 41:19
  SPEAKER_0: Because you can say things to it like, "Okay, now show me an example code that uses that option."
• 41:24
  SPEAKER_0: If you're using Claude Sonnet, you can say, "Show me the interactive prototype of that option."
• 41:29
  SPEAKER_0: All of that.
• 41:30
  SPEAKER_0: So that research stuff happens more quickly for me.
• 41:34
  SPEAKER_0: There's a whole bunch of those smaller productivity boosts.
• 41:37
  SPEAKER_0: The bigger one, the more interesting one for me, is I can take on much more ambitious project because I'm no longer limited to the things that I already know all of the trivia about.
• 41:48
  SPEAKER_0: And I feel like this is one of the most important aspects of all of this is if you want to program in Python or JavaScript or Go or Bash or whatever, there's a baseline of trivia that you need to have at the front of your mind.
• 42:01
  SPEAKER_0: You need to know how for loops work and how conditionals work and all of that kind of stuff.
• 42:05
  SPEAKER_0: And so I think there is a limit on the number of programming languages most people can work in.
• 42:09
  SPEAKER_0: I've found personally, I cap out about four or five programming languages, and if I want to start using another one, there's potentially a month-long spin-up for me to start, and that means I won't do it right.
• 42:22
  SPEAKER_0: Why would I use Go to solve a problem if I have to spend a month spinning up on Go when I could solve it with Python today?
• 42:29
  SPEAKER_0: That is gone.
• 42:30
  SPEAKER_0: I am using a much wider range of programming languages and tools right now because I don't need to know how for loops in Go work.
• 42:38
  SPEAKER_0: I need to understand the higher-level concept of Go like memory management and Go routines and all of that kind of stuff, but I don't have to memorize the trivia.
• 42:47
  SPEAKER_0: So given that, I've actually shipped Go code to production, despite not being a Go programmer just six months ago, that's been running happily every day.
• 42:56
  SPEAKER_0: And it has unit tests and it has continuous integration and continuous deployment and all of the stuff that I think is important for code.
• 43:03
  SPEAKER_0: And I could do that because the language model could fill in all of those little trivia bits for me.
• 43:09
  SPEAKER_1: This episode is sponsored by TLDR.
• 43:11
  SPEAKER_1: TLDR is a free daily newsletter covering the most interesting stores in tech startups and programming.
• 43:16
  SPEAKER_1: Join more than 1 million readers and sign up at TLDR.tech.
• 43:20
  SPEAKER_1: That is TLDR.tech.
• 43:23
  SPEAKER_1: I sometimes dread going back to certain side projects where it takes me a while to spin up and remember and it's in a language or an outdated framework that I just don't want touch.
• 43:33
  SPEAKER_1: And like what you said, the confidence is higher and I can actually just paste parts in the ChatGPT or turn on GitHub Copilot.
• 43:40
  SPEAKER_1: And I know what good looks like, so I think when you know that, even in a different language-
• 43:48
  SPEAKER_0: You need to have that experience.
• 43:48
  SPEAKER_0: If I was a brand new programmer, I don't think I'd be using it to write Go despite not knowing Go.
• 43:53
  SPEAKER_0: But I've got 20 years of experience.
• 43:55
  SPEAKER_0: I can read code that it's written in a language that I don't know very well and I can still make a pretty good evaluation of if that's doing what I need it to do, and if that looks like it's good.
• 44:07
  SPEAKER_0: And that's really important.
• 44:09
  SPEAKER_1: I guess there's an important disclaimer that the more you look at languages, as long as it's an imperative language, you can read it.
• 44:18
  SPEAKER_1: Some languages are just not as popular like Prolog and SML and some of these really declarative-
• 44:25
  SPEAKER_0: Yeah.
• 44:25
  SPEAKER_0: I would not trust myself to just look at Prolog code that it had written me and make a judgment as to whether that was good Prolog code.
• 44:32
  SPEAKER_0: But I feel like I can do that with languages Go and Rust.
• 44:35
  SPEAKER_1: Yeah.
• 44:36
  SPEAKER_1: So with that, I think it's good.
• 44:39
  SPEAKER_1: By the way, thanks for sharing.
• 44:40
  SPEAKER_1: I think it's great to see that you are getting productivity, but it also took a lot of work.
• 44:44
  SPEAKER_1: I think a big takeaway for me would be, anyone who's trying out, is put in the work and experiment to figure out what workflow works for yourself and that there's just no answers.
• 44:55
  SPEAKER_1: I think you've been experimenting a lot more than most people have and still sounds like it's a work in progress.
• 45:03
  SPEAKER_0: Oh, completely.
• 45:04
  SPEAKER_1: But with this, I really want to touch on misconceptions and doubts.
• 45:08
  SPEAKER_1: They might not be misconceptions.
• 45:09
  SPEAKER_1: They're doubts and questions that a lot of people have about these tools.
• 45:13
  SPEAKER_0: Let's talk about resistance a little bit because I feel like I see so much resistance to this, and it's a very natural and very understandable thing.
• 45:20
  SPEAKER_0: This stuff is really weird.
• 45:22
  SPEAKER_0: It's weird and it is uncomfortable and the ethics around it are so murky.
• 45:27
  SPEAKER_0: These models were trained on vast quantities of unlicensed copyrighted data.
• 45:32
  SPEAKER_0: And whether or not that's legal, and I'm not a lawyer, I'm not going to go into that.
• 45:36
  SPEAKER_0: The ethics of that, especially when you look at things like image models like stable diffusion, which are now being used when you would've commissioned an artist instead.
• 45:48
  SPEAKER_0: And they were trained on that artist's work.
• 45:50
  SPEAKER_0: I don't care if that's legal.
• 45:51
  SPEAKER_0: That's blatantly unfair, if something trained on the work and is now-
• 45:57
  SPEAKER_1: There is one person who wrote just this, that they tried it out, didn't work that well, plus they don't want to use it because they disagree fundamentally with this.
• 46:03
  SPEAKER_0: And honestly, I respect that position.
• 46:06
  SPEAKER_0: I've compared it to being vegan in the past.
• 46:10
  SPEAKER_0: That veganism, I think there's a very strong argument for why you should be a vegan.
• 46:15
  SPEAKER_0: And I understand that argument and I'm not a vegan, so I have made that personal ethical choice.
• 46:20
  SPEAKER_0: And all of this stuff does don't come down to personal ethical choices.
• 46:23
  SPEAKER_0: If you say, "I am not going to use these models until somebody produces one that was trained on entirely licensed data," I absolutely respect that.
• 46:33
  SPEAKER_0: I've not made that decision myself.
• 46:37
  SPEAKER_0: And for the code stuff, it's basically trained on every piece of open source code they could get on.
• 46:43
  SPEAKER_0: But it is ignoring the license terms, the GPM licenses that say attribution is important.
• 46:48
  SPEAKER_0: You can't attribute what comes out of a model.
• 46:50
  SPEAKER_0: It's been scrambled with everything else.
• 46:52
  SPEAKER_0: So yeah, the ethical concerns, I completely respect.
• 46:57
  SPEAKER_0: It's scary.
• 46:58
  SPEAKER_0: It is scary when you think, okay, I earn a very good salary because I have worked through the trivia of understanding Python and JavaScript and I'm better at that trivia than most other people.
• 47:08
  SPEAKER_0: And now you've got this machine that comes along and it's better at the trivia than I am.
• 47:15
  SPEAKER_0: The things that I "know," knows in scare quotes, that is disconcerting.
• 47:21
  SPEAKER_0: And I feel like there's a pessimistic and an optimistic way of taking this on.
• 47:25
  SPEAKER_0: The pessimistic way is saying, "Okay, I need to go into the trades.
• 47:30
  SPEAKER_0: I need to learn plumbing because my job is not going to exist in a few years' time."
• 47:35
  SPEAKER_0: The optimistic version, the version I take on, is I can use these tools better than anyone else for programming.
• 47:41
  SPEAKER_0: I can take my existing programming knowledge and when I combine it with these tools, I will run circles around somebody who's never written a line of code in their life and is trying to build an iPhone app using ChatGPT.
• 47:53
  SPEAKER_0: I can just do this stuff better.
• 47:55
  SPEAKER_0: So we've essentially got these tools that are... They're actually power user tools.
• 48:00
  SPEAKER_0: You have to put a lot of work into mastering them.
• 48:03
  SPEAKER_0: And when you've got that, when you combine expertise in using tools with expertise in a subject matter, you can operate so far above other people and the competitive advantage you get is enormous.
• 48:14
  SPEAKER_0: That's something that actually does worry me most about the resistance is I like people who are resisting this stuff.
• 48:21
  SPEAKER_0: I like that they're not falling for the hype.
• 48:23
  SPEAKER_0: I like that they care about the ethics of it.
• 48:25
  SPEAKER_0: I like that they're questioning it.
• 48:27
  SPEAKER_0: It would upset me if that put them at a serious professional advantage over the next few years as other people who don't share their ethics start being able to churn out more stuff.
• 48:39
  SPEAKER_0: It's like if you were to say, "I don't don't like search engines."
• 48:44
  SPEAKER_0: I'm never going to search for an answer to my programming problem."
• 48:46
  SPEAKER_0: That would set you back enormously right now.
• 48:49
  SPEAKER_0: And I feel like it's in a similar kind of space to that.
• 48:53
  SPEAKER_1: Yeah.
• 48:53
  SPEAKER_1: And so another opinion I hear a lot is, "Well, it seems like this whole technology is plateauing."
• 49:03
  SPEAKER_1: If we look at the past 18 months, ChatGPT-4 is... Okay, Claude might be a little bit better.
• 49:08
  SPEAKER_1: Sonnet.
• 49:09
  SPEAKER_1: Okay, cool.
• 49:09
  SPEAKER_1: But let's ignore that for just a second.
• 49:12
  SPEAKER_1: GitHub Copilot hasn't changed all that much.
• 49:15
  SPEAKER_1: So I do see a sense, especially for people who are managing engineers and they're also playing with this tool, saying, "Well, it sounds like this is what it's going to be.
• 49:25
  SPEAKER_1: We just use it as is.
• 49:27
  SPEAKER_1: Is this all?"
• 49:29
  SPEAKER_1: You're more in the weeds.
• 49:30
  SPEAKER_1: Do you see improvements or drastic improvements or little improvements?
• 49:36
  SPEAKER_0: That's a really interesting question.
• 49:37
  SPEAKER_0: From my perspective, I'd welcome a plateau at this point.
• 49:41
  SPEAKER_0: It's been a bit exhausting keeping up with the stuff over the last two years.
• 49:45
  SPEAKER_0: I feel like if there were no improvement, if what we have today is what we're stuck with for the next two years, it would still get better because we'd all figure out better ways to use it.
• 49:57
  SPEAKER_0: One of my favorite advances in language models is this thing called chain of thought prompting.
• 50:01
  SPEAKER_0: This is this thing where if you say to a language model, "Solve this puzzle," it'll often get it wrong.
• 50:06
  SPEAKER_0: And if you say, "Solve this puzzle, think step by step," and it'll then say, "Okay.
• 50:11
  SPEAKER_0: Step one, this.
• 50:12
  SPEAKER_0: Step two, step three."
• 50:13
  SPEAKER_0: And often it'll get it right.
• 50:15
  SPEAKER_0: And the wild thing about chain of thought prompting is that it was discovered against GPT-3 about a year after GPT-3 came out was an independent research paper that was put out saying, "Hey, it turns out, take this model to say think step by step and it gets better at all of this stuff."
• 50:31
  SPEAKER_0: Nobody knew that.
• 50:32
  SPEAKER_0: The people who built GPT-3 didn't know that.
• 50:33
  SPEAKER_0: It was an independent discovery.
• 50:35
  SPEAKER_0: We've had quite a few examples like this.
• 50:38
  SPEAKER_0: So if we are in a plateau, then I think we'll still get lots of advances from just people figuring out better ways to use the tooling.
• 50:48
  SPEAKER_0: A lot of this also comes down to whether or not you buy into the whole AGI thing.
• 50:51
  SPEAKER_0: So much of the mainstream-
• 50:58
  SPEAKER_1: That's a [inaudible 00:50:59] here.
• 50:59
  SPEAKER_0: Right.
• 50:59
  SPEAKER_0: It's like Tesla self-driving cars.
• 51:00
  SPEAKER_0: The CEOs of these companies go and say, "We're going to have AGI in two years' time.
• 51:05
  SPEAKER_0: It's coming.
• 51:06
  SPEAKER_0: Nobody will ever work again," which helps you raise a lot of money.
• 51:10
  SPEAKER_0: It scares me.
• 51:15
  SPEAKER_0: I'm not convinced that human economies will work if all knowledge work is replaced by AI.
• 51:20
  SPEAKER_0: And it also gives a very unrealistic idea of what these things can do, because fundamentally-
• 51:24
  SPEAKER_1: Well, and don't forget, it's also happening with software engineers.
• 51:26
  SPEAKER_1: There are companies out there whose pitch is we will replace software engineers with AI engineers, which is a very direct... Although I'm now starting to see a pattern of how this is really good for fundraising because it means a lot of potential market, and don't forget that that's who they're talking to.
• 51:44
  SPEAKER_1: And once they raise the money, they have that money, they can then operate.
• 51:48
  SPEAKER_1: And often, like in this case with Cognition AI, their claims are toned down to the point of it's pretty much GitHub Copilot.
• 51:57
  SPEAKER_1: But it is scary, because you see it in the mainstream media everywhere.
• 52:02
  SPEAKER_1: I think someone said, "We are replacing our own jobs as software engineers."
• 52:07
  SPEAKER_1: And as you said, I think it's the first time I've seen that written in the press.
• 52:11
  SPEAKER_1: Maybe this happened before I was born, but not recently.
• 52:16
  SPEAKER_0: It's funny, isn't it?
• 52:18
  SPEAKER_0: Who would've thought that AI would come for the lawyers and the software engineers and the illustrators and all of these things that normally you don't think of as being automatable?
• 52:28
  SPEAKER_0: But yeah, so the AGI thing, that leads to lots of disappointment.
• 52:31
  SPEAKER_0: People are like, "Yeah, well, I asked it this dumb logic puzzle and it got it wrong."
• 52:36
  SPEAKER_0: But it also ties into science fiction.
• 52:40
  SPEAKER_0: Everyone thinks about the Matrix and Terminator and all of that kind of stuff.
• 52:45
  SPEAKER_0: Honestly, the key problem here is these things can talk now.
• 52:49
  SPEAKER_0: They can imitate human speech.
• 52:53
  SPEAKER_0: And throughout human society, being able to write well and convincingly has always been how we evaluate intelligence.
• 52:59
  SPEAKER_0: But these things are not intelligent at all, but they can write really well.
• 53:03
  SPEAKER_0: They can produce very convincing text, which throws everyone off.
• 53:07
  SPEAKER_0: So yeah, if you are captured by the AGI hype, then I think, yeah, I think we're going to have a plateau.
• 53:15
  SPEAKER_0: I'd be very surprised if we had anything that was AGI.
• 53:17
  SPEAKER_0: Like I said, I have not been sold that this is a net win for humanity.
• 53:23
  SPEAKER_0: I don't know how society would cope with that.
• 53:26
  SPEAKER_0: But what we are seeing is incremental improvements.
• 53:30
  SPEAKER_0: Claude 3.5 sonnet is a substantial incremental improvement over GPT-4o and Claude 3 Opus.
• 53:40
  SPEAKER_0: The interesting thing about Claude 3.5 Sonnet is that it's named sonnet because their previous Claude 3 had three levels.
• 53:47
  SPEAKER_0: There was Haiku, Sonnet, and Opus.
• 53:48
  SPEAKER_0: Haiku was the cheap one.
• 53:50
  SPEAKER_0: Sonnet in the middle.
• 53:50
  SPEAKER_0: Opus was the really fancy one.
• 53:53
  SPEAKER_0: They have said they're going to release Haiku 3.5, which will be cheap and amazing, and Opus 3.5, which is going to be a step up from Sonnet.
• 54:01
  SPEAKER_0: I try to ignore the, "It's coming soon."
• 54:04
  SPEAKER_0: Those ones I am excited about in terms of it's coming soon.
• 54:08
  SPEAKER_0: But yeah, so if you are buying into the AGI stuff, then I don't buy into it.
• 54:13
  SPEAKER_0: I don't think you get to AGI from auto-completing sentences, no matter how good you are at auto-completing sentences.
• 54:20
  SPEAKER_0: And then, yeah, in terms of the plateau, incremental improvement is enough for me.
• 54:28
  SPEAKER_0: The models I've got right now, I want them to be faster.
• 54:33
  SPEAKER_0: Yeah.
• 54:34
  SPEAKER_1: If you look through back through history, I'm a little bit skeptical to believe that suddenly fundamental things would change in the software industry.
• 54:49
  SPEAKER_1: People sometimes project that this time it will be very different.
• 54:53
  SPEAKER_1: And again, there's always innovation, but looking back, we've always had innovation.
• 54:57
  SPEAKER_1: We've had some new technologies and then incremental improvement.
• 55:00
  SPEAKER_1: Pattern matching, that will be logical.
• 55:03
  SPEAKER_1: Obviously, there's black swan events.
• 55:06
  SPEAKER_1: Who could have seen COVID come?
• 55:08
  SPEAKER_1: Or this is also a breakthrough.
• 55:09
  SPEAKER_1: But I think there's a part of, we're not just in a vacuum, there's not just one event.
• 55:15
  SPEAKER_1: And AI has been predicted to be around the corner by different people since the start of computing, really, to be fair.
• 55:26
  SPEAKER_0: Something I think about a lot is the impact of TikTok and YouTube on professional video creation.
• 55:33
  SPEAKER_0: The iPhone is... This is a really great video camera, and TikTok and YouTube have meant that you can now publish videos to the entire world.
• 55:42
  SPEAKER_0: And that has not killed off professional video.
• 55:46
  SPEAKER_0: People who work professionally in that industry, they're doing fine.
• 55:49
  SPEAKER_0: What's happened is millions of people who would never have even dreamed of trying to learn to stand in front of a camera or to operate that equipment are now publishing different kinds of content online.
• 56:02
  SPEAKER_0: My ideal version of the AI programming thing is I want the number of people who can do basic programming to go up by an order of magnitude.
• 56:11
  SPEAKER_0: I think every human being deserves to be able to automate dull things in their lives with a computer.
• 56:18
  SPEAKER_0: And today, you almost need a computer science degree just to automate a dull thing in your life with a computer.
• 56:23
  SPEAKER_0: That's the thing which language models I think are taking a huge bite out of.
• 56:29
  SPEAKER_0: So there is a version of that where the demand for professional software engineers goes down because the more basic stuff can be done by other things.
• 56:37
  SPEAKER_0: The alternative version of that is the thing where, because a professional software engineer can now do five times the work that they used to do maybe, two times, five times, whatever it is, that means that companies that wouldn't have built custom software now do, which means that the number of jobs that software engineers goes up.
• 56:53
  SPEAKER_0: A company that would never have built its own custom CRM for their industry, because you'd have to hire 20 people and wait six months, can now do it with five people and two months.
• 57:02
  SPEAKER_0: And that means that-
• 57:03
  SPEAKER_0: Now do it with five people and two months, and that means that that's now feasible for them.
• 57:05
  SPEAKER_0: And those five people are still getting paid very well.
• 57:08
  SPEAKER_0: It's just that the value that they provide to companies has gone up, so that's the demand curve I'd like to see.
• 57:17
  SPEAKER_1: Well, and also don't forget, one thing that we do talk about, or I think it's kind of a common knowledge, correct me if it's wrong, but code equals liability.
• 57:26
  SPEAKER_1: The more code you have, the more liability you have.
• 57:28
  SPEAKER_1: And one thing, just what we're seeing is more code will be generated.
• 57:32
  SPEAKER_1: And at some point, I just think about this thing.
• 57:34
  SPEAKER_1: Have you worked at a company or a team where you just had less-experienced developers, one or two years experience, and you leave them for a while?
• 57:42
  SPEAKER_1: Or you might have seen-
• 57:44
  SPEAKER_0: Oh, yeah.
• 57:44
  SPEAKER_1: And then what happens, right?
• 57:46
  SPEAKER_1: Fast-forward to two years.
• 57:47
  SPEAKER_1: You don't add any one experience.
• 57:51
  SPEAKER_1: Usually, my observation is it's you get spaghetti code, it's a mess.
• 57:55
  SPEAKER_1: It's hard to do.
• 57:56
  SPEAKER_1: And then you pull in some people with more experience who look around, they point out some seemingly simple changes that are not that simple for the people, they simplify things, you might delete a lot of code, and then all will be good in the world, or those people will get more experience.
• 58:12
  SPEAKER_1: But I do think about this part where a year in, everything still seems to be fine, right?
• 58:18
  SPEAKER_1: The CEO at the company is like, "Oh, this team's shipping quickly.
• 58:21
  SPEAKER_1: People are enthusiastic."
• 58:23
  SPEAKER_1: And my sense is that there should be a demand, and again, I'm curious to hear your thoughts on this, but engineers who can go into the generated code and, for example, explain reason, even when the machine fails to explain this complicated mumble-jumble, or just say, "We're going to delete all of this, and it makes sense.
• 58:42
  SPEAKER_1: I'm confident I can tell you why I'm doing this."
• 58:46
  SPEAKER_0: Right.
• 58:46
  SPEAKER_0: And that's what I expect.
• 58:48
  SPEAKER_0: That's the skill that you need.
• 58:49
  SPEAKER_0: It turns out, the typing code and remembering how full-loops work, that's the piece of our jobs that has been devalued, right?
• 58:56
  SPEAKER_0: Remembering that sort of trivia and typing really quickly.
• 58:59
  SPEAKER_0: Nobody cares if you can type faster than anyone else anymore.
• 59:01
  SPEAKER_0: That's not a thing.
• 59:03
  SPEAKER_0: But the systems thinking and evaluating, a skill that I think is really important right now is QA in terms of just the old fact.
• 59:11
  SPEAKER_0: Manual testing, being able to take some code and really hammer away at it and make sure that it does exactly what it needs to do, combined with automated testing, combined with system design and prioritization, there's so much to what we do that isn't just typing code on a keyboard.
• 59:26
  SPEAKER_0: And those are the skills.
• 59:28
  SPEAKER_0: The thing is, language models can do a lot of this stuff, but only if you ask the right questions of them, right?
• 59:34
  SPEAKER_0: If you ask a language model to write five paragraphs on how you should refactor your microservices, maybe it'll do an okay job, but who's going to know to even pose that question, and who's going to know how to evaluate what it says?
• 59:48
  SPEAKER_0: So those decisions, these things, I don't think you should ever have them make decisions for you.
• 59:52
  SPEAKER_0: I think you should use them as tools to support the decisions that you're making.
• 59:57
  SPEAKER_0: It's one of the reasons I love saying give me options for X. And that's what we become, right?
• 1:00:01
  SPEAKER_0: Software engineers, we are the people making the high-level design decisions, the people evaluating what's going on.
• 1:00:08
  SPEAKER_0: I don't think you should ever commit a line of code that the language model wrote if you don't understand it yourself.
• 1:00:12
  SPEAKER_0: That's sort of my personal line that I draw.
• 1:00:16
  SPEAKER_0: And, yeah.
• 1:00:17
  SPEAKER_0: So I do not feel threatened as a software engineer.
• 1:00:19
  SPEAKER_0: And honestly, partly as a software engineer who's got good with this stuff, I really don't feel threatened by it.
• 1:00:25
  SPEAKER_0: But just generally, I think the bits of my job that these tools will accelerate, there are a whole bunch of bits of the job that accelerate, some of which are a bit tedious, some of which are kind of interesting, but it gives me so much more scope to take on more exciting problems overall.
• 1:00:43
  SPEAKER_1: I love it.
• 1:00:44
  SPEAKER_1: And if you could offer advice to two different groups of people, so two separate pieces, but experience engineers like yourself in terms of put in the years' worth across different stacks, and also to less-experienced engineers who they're already working inside the industry, but obviously, they're not at the level just yet.
• 1:01:04
  SPEAKER_1: What would you suggest to them to make the most out of these tools, or just to make themselves most more future-proof, if you will?
• 1:01:13
  SPEAKER_0: My universal advice is always to have side projects on the go, which doesn't necessarily work for everyone.
• 1:01:19
  SPEAKER_0: If you've got a family and a demanding job and so forth, it can be difficult to carve those out.
• 1:01:25
  SPEAKER_0: A trick I've used at companies in the past, I love advocating for internal hack days, saying let's once a quarter have everyone spend a day or two days working on their own projects.
• 1:01:36
  SPEAKER_0: That kind of stuff can be great.
• 1:01:37
  SPEAKER_0: Good employers should always be able to leave a little bit of wiggle room for that sort of exploratory programming, but some employers don't.
• 1:01:46
  SPEAKER_0: But if you can get that, that's amazing.
• 1:01:48
  SPEAKER_0: If you're earlier in your career, people in their 20s can normally get away with a lot of side projects because they have a lot less going on.
• 1:01:55
  SPEAKER_0: It's like when I'm managing people, I don't like people working super- long hours and all of that.
• 1:02:01
  SPEAKER_0: It's hard to talk a 22-year-old out of that.
• 1:02:04
  SPEAKER_0: That's just sort of how people are wired earlier in their careers, so take advantage of that while you can.
• 1:02:09
  SPEAKER_0: But, yeah.
• 1:02:09
  SPEAKER_0: I feel like my personal weblog, I'm using all sorts of weird AI tools to hack on that because the stakes could not be lower, right?
• 1:02:19
  SPEAKER_0: A bug in that, it'll break a page and I'll fix it.
• 1:02:22
  SPEAKER_0: So that's where I've been using, there's a thing called GitHub Copilot Workspaces that they've just started trialing.
• 1:02:28
  SPEAKER_1: Yeah.
• 1:02:28
  SPEAKER_1: Yeah.
• 1:02:28
  SPEAKER_1: It's [inaudible 01:02:30].
• 1:02:29
  SPEAKER_1: You're in [inaudible 01:02:30].
• 1:02:31
  SPEAKER_0: Yeah.
• 1:02:31
  SPEAKER_0: And I've added four or five features to my blog using that, some of them in meetings with people as a demo.
• 1:02:36
  SPEAKER_0: I'm like, "Oh, let's show you this tool.
• 1:02:38
  SPEAKER_0: I'm going to add auto-complete to the tags on my blog."
• 1:02:41
  SPEAKER_0: And I did that last week.
• 1:02:42
  SPEAKER_0: And so I'm using my blog as a sort of fun exploration space for some of that kind of thing.
• 1:02:48
  SPEAKER_0: But, yeah.
• 1:02:49
  SPEAKER_0: So if you can afford to do a side project with these tools, and set yourself a chance to write every line of code, have these tools write that code for you, I think that's a great thing you can do.
• 1:02:59
  SPEAKER_0: If you can't afford side projects, both of the best models are now free, like GPT-4o with OpenAI-
• 1:03:09
  SPEAKER_1: Yeah.
• 1:03:09
  SPEAKER_1: Crazy.
• 1:03:10
  SPEAKER_0: ... and Claude 3.5's on it.
• 1:03:10
  SPEAKER_0: Now, you have to log in, you might have to give them a phone number, but you can use a free account with them.
• 1:03:15
  SPEAKER_0: Use those, and just throw questions at them.
• 1:03:20
  SPEAKER_0: And sometimes, have a question where you think it definitely won't get this and throw that in, because that's useful information.
• 1:03:25
  SPEAKER_0: Throw in basic things.
• 1:03:26
  SPEAKER_0: Just work with them that way.
• 1:03:28
  SPEAKER_0: I think that's definitely worthwhile.
• 1:03:30
  SPEAKER_0: And the Claude 3.5 Artifacts thing is just so much fun.
• 1:03:34
  SPEAKER_0: The other day, I wanted to add a box shadow to a thing on a page, and I'm like, "What I really need is I need a sort of very light sort of subtle box shadow."
• 1:03:44
  SPEAKER_0: And then I was halfway through prompting Claude to that, and said, "Actually, build me a tool.
• 1:03:48
  SPEAKER_0: Build me a little tool," I think I said, "Where I can twiddle with the settings."
• 1:03:52
  SPEAKER_0: That was my prompt.
• 1:03:53
  SPEAKER_0: "Let me twiddle with the settings in the box shadow," and it built me this little interactive thing with a box shadow, and sliders with the different settings, and a copy-and-paste CSS thing.
• 1:04:01
  SPEAKER_0: And if I'd spent an extra 15 seconds on it, I could have found a tool that existed on Google, but it was faster to get Claude to build me a custom tool on demand.
• 1:04:11
  SPEAKER_0: Because if you're on a Google search, you have to evaluate the answers.
• 1:04:13
  SPEAKER_0: You get four back, and then you click through, and all that.
• 1:04:16
  SPEAKER_0: No.
• 1:04:16
  SPEAKER_0: No.
• 1:04:16
  SPEAKER_0: I know what I want.
• 1:04:17
  SPEAKER_0: So do that, right?
• 1:04:18
  SPEAKER_0: That's just wildly-
• 1:04:20
  SPEAKER_1: No.
• 1:04:20
  SPEAKER_0: ... entertaining.
• 1:04:21
  SPEAKER_1: I feel this is of what you're saying.
• 1:04:22
  SPEAKER_1: It's like, "Yeah."
• 1:04:23
  SPEAKER_1: I mean, it's easier said than done, but experimenting.
• 1:04:26
  SPEAKER_1: And I think your blog, which we're going to link in the show notes, is just a really good example.
• 1:04:31
  SPEAKER_1: I did find myself a little bit re-energized reading how much weird stuff you're doing.
• 1:04:36
  SPEAKER_1: Sorry for the-
• 1:04:37
  SPEAKER_0: That's the other thing.
• 1:04:38
  SPEAKER_0: It's got to be fun, right?
• 1:04:39
  SPEAKER_0: That-
• 1:04:40
  SPEAKER_1: Yeah. But I-
• 1:04:40
  SPEAKER_0: ... one of the things people miss in this space.
• 1:04:40
  SPEAKER_1: I can see that you're having fun with it.
• 1:04:44
  SPEAKER_1: And again, thanks for sharing, because you put it out there.
• 1:04:46
  SPEAKER_1: I think that's another thing, that.
• 1:04:48
  SPEAKER_1: But honestly, with these tools, it's a bit easier to write it up as well, so I think-
• 1:04:52
  SPEAKER_0: But, yeah.
• 1:04:53
  SPEAKER_1: ... it's helpful advice.
• 1:04:54
  SPEAKER_0: They're funny.
• 1:04:55
  SPEAKER_0: This is a crucial thing.
• 1:04:56
  SPEAKER_0: These things are absolutely hilarious.
• 1:04:58
  SPEAKER_0: And sometimes, they can write a joke that's good, but that's not what makes them funny.
• 1:05:03
  SPEAKER_0: It's trying out weird dystopian things, trying something you didn't think would work and having it work.
• 1:05:09
  SPEAKER_0: I use the voice mode.
• 1:05:11
  SPEAKER_0: I used to do prank phone calls to my dog, so I'll be like, "Hey, ChatGPT, I need to give my dog a pill covered in peanut butter.
• 1:05:19
  SPEAKER_0: I need you to pretend to be from the Government Department of Peanut Butter and make up an elaborate story about why she has to have it.
• 1:05:24
  SPEAKER_0: Now, go."
• 1:05:25
  SPEAKER_0: And it does it, and I hold the speaker up to my dog, and it's just really, really amusing.
• 1:05:31
  SPEAKER_0: So stuff like that is so much fun.
• 1:05:35
  SPEAKER_0: For a while, for a twist into my prompts, I'm like, "Answer this," and [inaudible 01:05:40] would say, "Oh.
• 1:05:40
  SPEAKER_0: And pretend you're a golden eagle and use golden eagle analogies."
• 1:05:42
  SPEAKER_0: And they would say, "Well, if you're soaring above the competition."
• 1:05:46
  SPEAKER_0: Stupid things like that, right?
• 1:05:48
  SPEAKER_0: You can get it to rap kind of, and it's awful.
• 1:05:52
  SPEAKER_0: Really absolutely appalling.
• 1:05:54
  SPEAKER_0: But with the voice mode, you can say, "Now, do a rap about that answer," and just it's cringe- worthy.
• 1:06:00
  SPEAKER_1: It is kind of wild how I don't really remember having a tool that, we're talking about programming here, but you can get it to do all these things within potentially even in the work context.
• 1:06:12
  SPEAKER_1: Just to throw it in there.
• 1:06:14
  SPEAKER_1: As you said, it is fun, so I like to look at that part of it.
• 1:06:19
  SPEAKER_1: So thank you for the insight, and let's end with some rapid questions in the end, if you're up for it?
• 1:06:24
  SPEAKER_0: Okay.
• 1:06:24
  SPEAKER_0: Sure.
• 1:06:25
  SPEAKER_1: So these are questions I'm going to ask, and you just throw out whatever comes up.
• 1:06:30
  SPEAKER_1: Could you recommend two or three books to people that you enjoyed reading?
• 1:06:34
  SPEAKER_0: Martin Kleppmann's book, Designing Data-Intensive Applications.
• 1:06:37
  SPEAKER_1: Yeah.
• 1:06:37
  SPEAKER_1: Here.
• 1:06:38
  SPEAKER_1: It's on my shelf, actually.
• 1:06:40
  SPEAKER_0: Absolutely incredible.
• 1:06:41
  SPEAKER_1: The Blue Sky team told me, Martin Kleppmann advises them, that this is the book they have all on their shelf, because this describes everything you need to know to build Blue Sky.
• 1:06:49
  SPEAKER_1: It's kind of amazing.
• 1:06:51
  SPEAKER_0: At Eventbrite, we had a book club, and one of the things we did with the book club, because nobody reads the book for book clubs, right?
• 1:06:57
  SPEAKER_0: It turns out that just doesn't work.
• 1:06:58
  SPEAKER_0: So what you do instead is you assign chapters to different people, and they have to provide a summary of the chapter at the book club.
• 1:07:04
  SPEAKER_0: So it's almost like you parallelize the act of reading the book.
• 1:07:09
  SPEAKER_0: That worked so well, and I think that was the best book that we did for that one.
• 1:07:16
  SPEAKER_1: And is there maybe a fiction book that you can recommend?
• 1:07:19
  SPEAKER_0: So my favorite genre of fiction is British wizards tangled up in old-school British bureaucracy.
• 1:07:26
  SPEAKER_0: So there's Charles Stross does the Laundry File Series, which is about sort of secret MI5s-style wizards.
• 1:07:34
  SPEAKER_0: There's the River of London series by Ben Aaronovitch, which are a metropolitan police officer who gets tangled up in magic.
• 1:07:40
  SPEAKER_0: I really enjoy those.
• 1:07:42
  SPEAKER_1: Oh, nice.
• 1:07:42
  SPEAKER_1: What's your favorite programming language and framework?
• 1:07:44
  SPEAKER_1: And you cannot say Django and Python.
• 1:07:47
  SPEAKER_0: Yeah.
• 1:07:47
  SPEAKER_0: Really putting me on the spot with this one.
• 1:07:49
  SPEAKER_1: Oh, yeah.
• 1:07:51
  SPEAKER_0: Okay.
• 1:07:51
  SPEAKER_0: JavaScript and no framework at all.
• 1:07:52
  SPEAKER_0: I love doing the vanilla JavaScript thing.
• 1:07:55
  SPEAKER_0: So I used to love jQuery, and now, document.
• 1:07:59
  SPEAKER_0: query, select all, and array.map and stuff, jQuery is built into browsers now.
• 1:08:03
  SPEAKER_0: You don't need an extra library.
• 1:08:05
  SPEAKER_1: It is kind of wild.
• 1:08:06
  SPEAKER_1: Yeah.
• 1:08:06
  SPEAKER_1: I remember that one when I used to do those.
• 1:08:08
  SPEAKER_1: I'm surprised.
• 1:08:09
  SPEAKER_1: Nice.
• 1:08:11
  SPEAKER_1: What's an exciting company that you're interested in and why?
• 1:08:17
  SPEAKER_0: So I'm going to plug Fly.io here, the hosting company, partly because they sponsor some of my work, but no.
• 1:08:24
  SPEAKER_0: Actually, completely independently of their sponsorship, I picked them to build my dataset Cloud SaaS platform on, because they're a hosting company that makes it incredibly easy to spin up secure containers as part of your infrastructure.
• 1:08:37
  SPEAKER_0: So basically, I was trying to build this stuff on top of Kubernetes, which is not easy to use.
• 1:08:41
  SPEAKER_0: And then I realized that Fly.io, their machines layer is effectively what you can do with Kubernetes, but with an API that actually makes sense and pricing that makes sense.
• 1:08:49
  SPEAKER_0: So I'm able to build out this SaaS platform where every one of my paying customers gets a private separate container running my software with its own encrypted volumes and all of that kind of thing.
• 1:08:59
  SPEAKER_0: And so I don't have to worry about data leaking from one container to another, and it scales to zero, so it scales to zero in between the quests and all of that kind of stuff, so yeah.
• 1:09:07
  SPEAKER_0: I'm really-
• 1:09:08
  SPEAKER_1: Awesome.
• 1:09:08
  SPEAKER_0: ... excited about Fly as a platform for specifically building that thing where you've got an open-source project and you want to run it for your customers.
• 1:09:16
  SPEAKER_0: Like paid hosting of open source, I feel like Fly is a really great platform for that.
• 1:09:21
  SPEAKER_1: Awesome.
• 1:09:22
  SPEAKER_1: Well, thanks very much.
• 1:09:22
  SPEAKER_1: It was great having you.
• 1:09:24
  SPEAKER_0: Cool.
• 1:09:24
  SPEAKER_0: This has been really fun.
• 1:09:25
  SPEAKER_0: Thanks a lot.
• 1:09:26
  SPEAKER_1: Thanks a lot to Simon for this.
• 1:09:28
  SPEAKER_1: If you'd like to find Simon online, you can do so on his blog, SimonWilson.net, and on social media, like X and Mastodon, all in the show notes below.
• 1:09:37
  SPEAKER_1: You can also check out his open-source projects, dataset, and LLM, which are also in the notes.
• 1:09:42
  SPEAKER_1: As closing, here are my top three takeaways from this episode.
• 1:09:46
  SPEAKER_1: Takeaway number one: if you're not using LLMs for your software engineering workflow, you are falling behind, so use them.
• 1:09:53
  SPEAKER_1: Simon outlined a bunch of reasons that hold back many devs from using these tools, from ethical concerns to energy concerns, but LLM tools are here to stay, and who those use them get more productive, so give yourself a chance with these.
• 1:10:06
  SPEAKER_1: Takeaway number two: it takes a ton of effort to learn how to use these tools efficiently.
• 1:10:11
  SPEAKER_1: As Simon put it, you have to put in so much effort to learn, explore, and experiment on how to use them, and there's no guidance, so you really need to put in the time and experimentation.
• 1:10:22
  SPEAKER_1: By the way, in a survey ran in the Pragmatic Engineer about AI tools, with about 200 software engineers responding, we saw some similar evidence.
• 1:10:30
  SPEAKER_1: Those who have not used AI tools for six months were more likely to be negative about the perception of these.
• 1:10:36
  SPEAKER_1: In fact, the very common feedback from engineers not using these tools was that they use it a few times, but it just didn't live with their expectations, and they just stopped using them.
• 1:10:44
  SPEAKER_1: I asked Simon how long it took him to get good at these tools, and he told me it just took a lot of time.
• 1:10:49
  SPEAKER_1: He couldn't put an exact number of months on it, but it just took a bunch of time and experimentation and figuring out if it works.
• 1:10:57
  SPEAKER_1: My third and final takeaway is that using local models to learn more about large language models is a smart strategy.
• 1:11:04
  SPEAKER_1: Running local models has two bigger benefits.
• 1:11:07
  SPEAKER_1: Number one, you figure out how to just do these, how to run models locally.
• 1:11:11
  SPEAKER_1: It's actually less complicated than one would think thanks to tools like Hugging Face that make downloading and running models a lot easier.
• 1:11:18
  SPEAKER_1: So just go and play around with them and see how a smaller model feels like.
• 1:11:23
  SPEAKER_1: The second benefit is that you learn a lot more about how large language models works because local models are just less capable, so they feel less magical.
• 1:11:32
  SPEAKER_1: Simon said how it's really useful to have a model hallucinate at you early because it helps you get better at the mental model of what it can do, and the local models do hallucinate wildly.
• 1:11:42
  SPEAKER_1: You'll also find some additional resources in the Pragmatic Engineer.
• 1:11:46
  SPEAKER_1: One of them is about RAG: Retrieval Augmented Generation.
• 1:11:50
  SPEAKER_1: This is an approach that Simon talks about in this episode.
• 1:11:53
  SPEAKER_1: It's a common building block for AI applications.
• 1:11:56
  SPEAKER_1: We did a deep-dive in the Pragmatic Engineer about this approach, and this is linked in the show notes below.
• 1:12:01
  SPEAKER_1: Also in the Pragmatic Engineer, we did a three-part series on AI tooling for software engineers, Reality Check.
• 1:12:07
  SPEAKER_1: We looked at how engineers are using these tools, what their perception is, what advice they have to use these tools more efficiently.
• 1:12:14
  SPEAKER_1: Personally, I cannot remember any developer tool or development approach that has been adopted so quickly by the majority of back-end and front-end developers in the first two years of its release like large language model tools have done so since 2022, so it's a good idea to not sleep on this topic.
• 1:12:31
  SPEAKER_1: And this marks the end of the first episode on the Pragmatic Engineer Podcast.
• 1:12:34
  SPEAKER_1: Thanks a lot for listening and watching.
• 1:12:36
  SPEAKER_1: If you enjoyed the episode, I'd greatly appreciate if you subscribed and left a review.
• 1:12:41
  SPEAKER_1: Thanks, and see you in the next one.