Latest / Coffee with Developers / Evals vs. Evil - AI and Package Security - Laurie Voss

Transcript

Evals vs. Evil - AI and Package Security - Laurie Voss · Coffee with Developers

• 0:01
  SPEAKER_00: We're back with another coffee with developers here live at Agent Con San Francisco.
• 0:06
  SPEAKER_00: No Silicon Valley.
• 0:08
  SPEAKER_00: It's close.
• 0:09
  SPEAKER_00: I'm back with a person I always like to have on the show is Lori Voss.
• 0:13
  SPEAKER_00: How are you doing today?
• 0:14
  SPEAKER_00: I'm doing great.
• 0:15
  SPEAKER_00: Thank you for having me.
• 0:16
  SPEAKER_00: You just gave a talk.
• 0:16
  SPEAKER_00: What do you talk about?
• 0:18
  SPEAKER_01: I was talking about evals and what they are and why you should do them and why they're difficult to get right.
• 0:24
  SPEAKER_00: So what are they and why should you do to use them?
• 0:30
  SPEAKER_01: The thing that I don't like about uh the ML industry is that it invents words uh that make simple concepts sound more complicated.
• 0:38
  SPEAKER_01: Evals are tests.
• 0:39
  SPEAKER_01: Okay.
• 0:40
  SPEAKER_01: That's what they are.
• 0:41
  SPEAKER_01: You can't write unit tests uh for LLMs because LLMs produce a different output every single time.
• 0:47
  SPEAKER_01: So if you're just doing you know string matching like you do in uh a regular unit test, it's going to fail all the time because your LLM produces a different output.
• 0:56
  SPEAKER_01: Uh so the solution is you get another LLM to look at the output of the first LLM and tell you whether or not it did a good job.
• 1:03
  SPEAKER_01: And that's an eval.
• 1:04
  SPEAKER_00: So it's like pair programming for LLMs.
• 1:07
  SPEAKER_01: Yes.
• 1:08
  SPEAKER_01: Usually you get usually you get like a bigger, stronger, slower LLM to judge the output of the smaller, faster LLM.
• 1:15
  SPEAKER_01: Uh, but not always.
• 1:17
  SPEAKER_00: Because I've seen that fail when people use the same model to review itself rather than like another model.
• 1:22
  SPEAKER_01: Yeah, that is uh a known limitation of LLMs is that they uh they enjoy outputs from themselves.
• 1:30
  SPEAKER_01: Uh if you get Sonnet to judge Sonnet, it will always think that it's slightly better than any other model.
• 1:35
  SPEAKER_01: So one of the bits of advice I gave in the talk was uh you should use a model from a different lab uh to measure uh your model.
• 1:43
  SPEAKER_01: So if you're using anthropic, you should get open AI to be your judge.
• 1:46
  SPEAKER_01: If you're using uh OpenAI, do the opposite.
• 1:49
  SPEAKER_00: Or basically get a free one to actually do it so you don't actually get burned out on tokens just by reviewing.
• 1:55
  SPEAKER_00: Absolutely, yeah.
• 1:57
  SPEAKER_00: So that to me is the biggest thing that I'm very much uh uh disenchanted about when it comes to the whole agentic world and like codes uh generated by AI.
• 2:06
  SPEAKER_00: That right now it seems to be the the uh the honeymoon period is over.
• 2:10
  SPEAKER_00: Now everybody realizes oh, electricity and machines cost money.
• 2:14
  SPEAKER_00: So all of a sudden it's like, oh yeah, it's it's not like free tokens by monthly or or yearly subscription, but you pay by token.
• 2:21
  SPEAKER_00: Yeah.
• 2:21
  SPEAKER_00: A lot of people are actually realizing they're spending much more money than they should.
• 2:24
  SPEAKER_01: Yeah.
• 2:25
  SPEAKER_01: I think there's there's a lot of sides to that.
• 2:29
  SPEAKER_01: Um I think uh one is that there are some people who uh are getting so much value out of AI, out of the acceleration of their coding by AI, for instance, that even paying per token at real costs is absolutely worth it to them.
• 2:46
  SPEAKER_01: Some people are are genuinely seeing value out of this stuff.
• 2:49
  SPEAKER_01: Um some people should frankly be switching to cheaper models.
• 2:54
  SPEAKER_01: Um the the the joy of of the open source models is that uh they are much, much cheaper to run and almost as good.
• 3:02
  SPEAKER_01: They uh they keep mysteriously catching up uh to to the main models through a process that they claim isn't distillation, but probably is.
• 3:12
  SPEAKER_01: Uh so you can get there a lot more cheaply than you used to.
• 3:16
  SPEAKER_01: Um but also frankly, a lot of people are gonna lose their shirts.
• 3:20
  SPEAKER_01: Like uh, you know, it's a bubble.
• 3:22
  SPEAKER_01: Uh and we've been through these before, you and I.
• 3:25
  SPEAKER_01: Uh we we know how these bubbles go.
• 3:29
  SPEAKER_01: Uh there's usually uh a strong residue of actual value left over after the bubble subsides.
• 3:36
  SPEAKER_01: Um, but some people do lose their shirts.
• 3:38
  SPEAKER_00: So we're here at AgentCon, and of course, the whole story is like everything is agentic, and agentic is the future, and it's gonna be completely different to what we've done before.
• 3:47
  SPEAKER_00: And if you're not part of it, you're gonna be basically falling behind immediately, and it's everything's gonna be horrible.
• 3:54
  SPEAKER_00: Uh, what's your thinking about that?
• 3:55
  SPEAKER_00: Like, what is the agentic movement right now, and are we there?
• 3:59
  SPEAKER_00: Is it that much of an innovation?
• 4:01
  SPEAKER_01: Um, I think the joke that I tell about this usually is whenever I hear somebody say that it's different this time, that's my number one signal that it's going to be the same this time.
• 4:11
  SPEAKER_01: Um we are uh certainly in an AI bubble, um, but there is a core of value there uh in a way that, for instance, I didn't think of there being a core of value in crypto.
• 4:24
  SPEAKER_01: Like I was never on the crypto train, but AI, I can see actual value coming out of it.
• 4:30
  SPEAKER_01: The analogy that I draw most often is uh to the introduction of the iPhone.
• 4:36
  SPEAKER_01: There was a lot of talk when the iPhone was introduced that this is how people are going to interact with computers now, websites are gonna be dead, everything's gonna be an app, and that isn't what happened.
• 4:45
  SPEAKER_01: But it's absolutely true that if you run a company these days, your company has either a mobile website or an app.
• 4:53
  SPEAKER_01: Uh so every company thinks about mobile in a way that they didn't prior to the introduction of the iPhone.
• 5:00
  SPEAKER_01: And every company is going to think about AI as part of their stack, as part of their offering uh after this bubble has passed.
• 5:09
  SPEAKER_01: That doesn't mean that everything is going to be an AI application.
• 5:11
  SPEAKER_01: It doesn't mean that everything is going to be an agent.
• 5:13
  SPEAKER_01: There's an awful lot of regular software that works just great.
• 5:17
  SPEAKER_01: Um, but uh there are a ton of applications for the things that LLMs are good at.
• 5:22
  SPEAKER_01: And LLMs are good at taking large quantities of unstructured data and turning it into more focused structured data.
• 5:30
  SPEAKER_01: Uh, and that's a thing that a lot of computer applications need to do.
• 5:34
  SPEAKER_01: Uh so I do think there's a core of value there that's gonna hang around.
• 5:39
  SPEAKER_00: I I totally agree, and that that's why I loved LLM's forge from the beginning.
• 5:43
  SPEAKER_00: But like then we got into the Gen AI thing, and we just create tons and tons of unstructured data to structure again.
• 5:50
  SPEAKER_00: So it feels like how do we work around the whole idea of AI slop?
• 5:54
  SPEAKER_00: And that comes in every form.
• 5:56
  SPEAKER_00: I mean, it's like horrible things on uh on social media, but it's also bad pull requests on open source projects.
• 6:02
  SPEAKER_01: I think uh it's a a real and genuine problem when people uh misuse AI.
• 6:10
  SPEAKER_01: Um and I would say that you're misusing an AI when you are expecting to get more out of it than you put into it.
• 6:18
  SPEAKER_01: Uh when people try to do that with like creative professions, uh when they do it with art, when they do it with music, when they do it with movies, um we can tell.
• 6:29
  SPEAKER_01: Like we could we can tell and we don't like it.
• 6:31
  SPEAKER_01: Uh like you know, I uh kids these days, like you know, six to ten-year-olds, like uh if they say that's AI, that's slang for that sucks.
• 6:41
  SPEAKER_00: Uh weirdly enough, even in German.
• 6:44
  SPEAKER_00: Yeah.
• 6:47
  SPEAKER_01: It turns out it turns out slang is universal.
• 6:50
  SPEAKER_01: Yeah.
• 6:51
  SPEAKER_01: Yeah.
• 6:52
  SPEAKER_01: Um so if you misuse AI, I think you produce slop, and I think uh we're going to have to put guardrails against it.
• 6:59
  SPEAKER_01: Um I think that's burning open source in particular right now, uh, because coding is sort of the first wave of things that people are uh really, really applying AI to.
• 7:10
  SPEAKER_01: Uh and open source projects are overwhelmed with slop PRs.
• 7:14
  SPEAKER_01: Uh I was doing a demo the other day where for my demo I needed a GitHub repo so that I could test an agent that's job was to manage a GitHub repo.
• 7:26
  SPEAKER_01: And so I made it as a public repo.
• 7:27
  SPEAKER_01: And as soon as I did, people's bots showed up and started trying to fix my fake code and my fake repo.
• 7:32
  SPEAKER_01: Oh wow.
• 7:33
  SPEAKER_01: Like minutes later.
• 7:36
  SPEAKER_01: Uh so it's it's a genuine problem that uh a lot of open source is struggling with, and I don't feel we have a good answer to yet.
• 7:43
  SPEAKER_00: Mind you, we always had that problem.
• 7:45
  SPEAKER_00: I I I I I remember pingback the other day on like uh on like WordPress sites and these kind of things, and that really, really quickly turned into like a spam a spam act access point.
• 7:56
  SPEAKER_00: It seems like you can't put an open system out on the web that attracts the the horrible bits first.
• 8:02
  SPEAKER_01: Yeah, you it is the it is the curse of the open web is that if it if it can be exploited, it will be exploited.
• 8:10
  SPEAKER_01: Uh and that means that sometimes we can't have nice things.
• 8:13
  SPEAKER_00: Yeah.
• 8:14
  SPEAKER_00: Talking of which, uh you used to be in a company called WDU, I think.
• 8:19
  SPEAKER_00: Um NPM.
• 8:21
  SPEAKER_00: So I like that when the logo was the wrong way around and people like WDU.
• 8:27
  SPEAKER_00: Um now security and especially uh supply chain attacks uh have like multiplied immensely since the agentic AI and well even just congenerated AI code and agentic AI.
• 8:41
  SPEAKER_00: How where are we in this situation?
• 8:43
  SPEAKER_00: Because it feels like there's a horror story every two days.
• 8:46
  SPEAKER_01: Um it does feel that way.
• 8:48
  SPEAKER_01: I uh I still have uh news alerts that go off every time NPM is mentioned in the news, uh, which is a very anxiety-inducing thing to still have, uh, because every day it's like a new worm has been birthed on NPM and is spreading across all of the NPM repositories, and everything is collapsing.
• 9:05
  SPEAKER_01: Part of it is uh a storm of identic-powered uh security research happening.
• 9:13
  SPEAKER_01: Um and the defenses against that have not caught up to the attackers.
• 9:18
  SPEAKER_01: We are seeing a temporary imbalancing of when the agents that find vulnerabilities and the agents that fix vulnerabilities are not being run at the same uh level of enthusiasm and volume.
• 9:31
  SPEAKER_01: Um uh but also there's uh a certain amount of underinvestment that has happened in the NPM uh uh ecosystem in general, um, in terms of uh guardrails and security and like defaults in the client and stuff like that.
• 9:48
  SPEAKER_01: Like there are, you know, one of the reasons that PNPM is popular is P NPM has changed the defaults of how NPM works uh to be a little bit more secure.
• 9:57
  SPEAKER_01: Um and as somebody who has an opinion about that, I feel like uh the time has come to change some of the defaults in the default NPM client um to make things uh a little bit harder.
• 10:10
  SPEAKER_00: That feels uh like a genuine thing that comes up all the time, though.
• 10:14
  SPEAKER_00: Like I feel every time when you say like, okay, we should do things in a safe manner and a secure manner, oh you're you're in the way of progress.
• 10:22
  SPEAKER_00: You're actually not trying to ship fast and fail fast, and but that comes back all the time.
• 10:27
  SPEAKER_00: Like I don't care if it created like 600 megabytes of JavaScript uh in my bytecoded app because I shipped it faster than the other people.
• 10:36
  SPEAKER_00: Do you think we need to run against the wall with that?
• 10:39
  SPEAKER_00: Or is it something that will keep that will it there always be a tension between innovation and security and privacy?
• 10:47
  SPEAKER_01: I think it's a really interesting question right now because one of the reasons uh I'm a big fan of frameworks and web frameworks in particular.
• 10:55
  SPEAKER_01: I've always been I've always spoken up for them, even when people say, oh, this framework slows you down and like adds a bunch of craft.
• 11:01
  SPEAKER_01: Like one of the reasons frameworks worked is because coders are expensive.
• 11:06
  SPEAKER_01: If you can find a way to speed up a coder, you save a lot more money than the uh money you lose by having a slow application.
• 11:15
  SPEAKER_01: Um but now coders have not got as expensive.
• 11:19
  SPEAKER_01: Coders are getting cheaper very, very rapidly because code is getting cheaper.
• 11:23
  SPEAKER_01: Could be the case that um we're going to there's going to be a period of of chaos that we're in right now, and then a rebalancing.
• 11:35
  SPEAKER_01: Okay.
• 11:35
  SPEAKER_01: Uh where we see things shake out at a higher velocity, um, but with guardrails in place and with checks in place that make it sustainable to work at that higher velocity.
• 11:48
  SPEAKER_00: I mean, it's also the it's sometimes such a self-fulfilling prophecy.
• 11:51
  SPEAKER_00: People want to work in a faster place to create more things without thinking, is this really necessary to do it?
• 11:58
  SPEAKER_00: Like, you know, like oh, the app needs to have a new feature every two days, or you're not gonna win product hunt.
• 12:04
  SPEAKER_01: Yeah, I went to um uh AI Engineer Miami uh a conference a couple of weeks ago, and uh more than one speaker, uh their solution to what do I do with this tide of slop is to just release slower.
• 12:17
  SPEAKER_01: Don't don't feel like you have to, you know, release fewer features, release, do slower reviews, throw more stuff away, uh, and do and ship fewer features.
• 12:27
  SPEAKER_01: Uh and it's a very it's a very tempting argument.
• 12:31
  SPEAKER_01: I can't I can't say that I agree.
• 12:33
  SPEAKER_01: I think I think going in my heart, going faster does feel better, and I think that we will figure it out.
• 12:41
  SPEAKER_01: But uh how we're gonna figure it out, what that answer is gonna be, is absolutely up in the air right now.
• 12:46
  SPEAKER_01: Uh, and there's gonna be a very painful transition period as we get there.
• 12:51
  SPEAKER_00: There is a joke going around that you can save tokens by hiring junior developers for the simple tasks.
• 12:57
  SPEAKER_00: Uh you're currently hiring, you hired a few people, and other people are just laying off by the truckload.
• 13:02
  SPEAKER_00: Like, what's going on with the with the with the software market in terms of jobs?
• 13:08
  SPEAKER_01: Um, I think uh anybody anybody laying off in a bubble um is probably doing it wrong.
• 13:16
  SPEAKER_01: Uh I think the job of a developer has changed very suddenly.
• 13:22
  SPEAKER_01: And uh the way that you become a developer has also changed.
• 13:26
  SPEAKER_01: Uh a lot of people think of the job of developer as, you know, you start as being a junior developer whose job is basically just to lay down code at somebody else's instruction, and then slowly you get a sense of architecture, and then slowly you get a sense of distributed systems, and then you get a sense of like security.
• 13:42
  SPEAKER_01: Uh that first step has collapsed.
• 13:45
  SPEAKER_01: That that job that you could get before, where your job was just to lay down code at somebody else's instruction, um, is going away because uh you can get an agent to do that.
• 13:55
  SPEAKER_01: Um, but the rest of those jobs uh still exist.
• 14:00
  SPEAKER_01: And uh you have to you have to sort of jump over that missing step to get there.
• 14:06
  SPEAKER_01: But there's no reason, to my mind, that you can't make architecture be the first thing that you learn as a software developer.
• 14:14
  SPEAKER_01: If coding is no longer a thing you need to learn, then learn architecture.
• 14:18
  SPEAKER_00: I mean, that's what a lot of companies say that are like AI A AI native, like TESL and people like that that say, like, okay, spec writing is becoming the new first thing to do, which is so hilarious because we were like the rogue developers in the past that made fun of the Java developers that wrote a spec first.
• 14:35
  SPEAKER_00: But it seems like if anything can be generated, writing a good spec, much like writing a good prompt, is the next skill to master.
• 14:42
  SPEAKER_01: Yeah, it's all like I said, it's earlier, it's all about turning a lot of material into a smaller amount of material.
• 14:48
  SPEAKER_01: That's what LLMs are good at doing.
• 14:50
  SPEAKER_01: Uh so if you want a really high-quality piece of software, you want a voluminous spec.
• 14:55
  SPEAKER_01: You want everything specified, you want to know absolutely everything this piece of software is gonna do, and your agent can boil it down into a nice compact piece of software.
• 15:06
  SPEAKER_00: It is interesting that the joke and when iPhones came out was to actually use autocompletion and see what nonsense comes out of it.
• 15:13
  SPEAKER_00: And that became the basis of every software we do right now.
• 15:16
  SPEAKER_01: Yes, indeed.
• 15:23
  SPEAKER_00: Just really bizarre, isn't it?
• 15:26
  SPEAKER_00: Now we're coming in September to uh to Silicon Valley and try to run the thing we do in Berlin.
• 15:31
  SPEAKER_00: Uh looking forward 10,000 developer event, and we want to make a very much focus of technical content only and also developers in the audience.
• 15:40
  SPEAKER_00: Do you think that's something that the that the valley needs right now?
• 15:43
  SPEAKER_00: Is this something that you missed?
• 15:46
  SPEAKER_01: I think uh one of the uh impossible to replicate things about Silicon Valley and about San Francisco uh is um the the uh unpredictable power of human connections.
• 16:04
  SPEAKER_01: Like you hear about how is this company started?
• 16:07
  SPEAKER_01: Oh, well, this guy used to live next door to this other guy, or you know, this lady used to go to the same climbing gym as that guy.
• 16:14
  SPEAKER_01: Uh and that's those connections are um they they are how cross-domain uh discoveries happen, they are how important people meet each other in their early days.
• 16:28
  SPEAKER_01: Um, it's having an in-person event in Silicon Valley is more important now than it's ever been.
• 16:34
  SPEAKER_00: Cool.
• 16:34
  SPEAKER_00: So I look forward to seeing you at hours, so that's gonna be good fun.
• 16:38
  SPEAKER_00: I'll see you there.
• 16:39
  unknown: Cool.
• 16:40
  SPEAKER_00: Well, thanks very much.
• 16:41
  SPEAKER_00: This was Coffee with Developers uh here at the Agent Confit Silicon Valley.
• 16:45
  SPEAKER_00: I leave you to your admiring uh crowd asking you questions.
• 16:49
  SPEAKER_00: And uh it's always a joy to have you.
• 16:51
  SPEAKER_00: I'm really looking forward to you.
• 16:52
  SPEAKER_00: Thanks for having me.
• 16:53
  SPEAKER_00: Thanks for being here.
• 16:54
  SPEAKER_00: See ya.
• 16:55
  SPEAKER_00: Bye bye.