Latest / The Ruby AI Podcast / RubyGems Security, AI Benchmarking, and Impatient Programming

Transcript

RubyGems Security, AI Benchmarking, and Impatient Programming · The Ruby AI Podcast

0:00
SPEAKER_00: Hey everybody.
0:01
SPEAKER_00: Welcome back to another episode of the Ruby AI Podcast.
0:04
SPEAKER_00: I am one of your hosts today, Valentino Stoll.
0:06
SPEAKER_01: I'm the other host.
0:07
SPEAKER_01: I'm Joe Leo.
0:08
SPEAKER_01: Valentino, do you know who you are looking at right now?
0:11
SPEAKER_01: You are looking at the newest.
0:14
SPEAKER_01: I know the mind reels.
0:15
SPEAKER_01: You're looking at the newest member of the Ruby Gems security on call team.
0:22
SPEAKER_01: Oh, you?
0:23
SPEAKER_01: That's me.
0:23
SPEAKER_01: Wow.
0:24
SPEAKER_01: Yeah.
0:25
SPEAKER_01: I just joined the team.
0:26
SPEAKER_01: I don't want to say when my next on-call is because I don't want people that are listening to like plan their attacks for my first day.
0:33
SPEAKER_01: But yeah, I'm going to join a rotation.
0:38
SPEAKER_01: Join a rotation.
0:39
SPEAKER_01: And so it's exciting to just be part of it.
0:42
SPEAKER_01: The truth is, you know, talking to Colby Swandale, and he's like, look, you know, this thing really runs pretty well.
0:47
SPEAKER_01: Like, hopefully, your job is really not is very infrequent that we need you, which is great.
0:52
SPEAKER_01: I mean, they obviously have done a tremendous amount of work, definitely in the last year as well.
0:58
SPEAKER_01: And we had plenty to say on the show about the RubyGems or the Ruby Central team and the transition that we still probably don't know everything about.
1:05
SPEAKER_01: But I have to say that they have a good operation for keeping that thing up and running.
1:10
SPEAKER_01: They have run books in place, they have all of this stuff to get people who are interested in just being a part of you know the maintenance or success of RubyGems to get involved.
1:20
SPEAKER_01: So yeah, I decided to not just stand on the sidelines and chirp, although that's you know something I like to do and jump in and get involved.
1:26
SPEAKER_01: That's fantastic to hear.
1:27
SPEAKER_00: Yeah.
1:27
SPEAKER_00: Fantastic to hear that it's so well built.
1:30
SPEAKER_00: I know.
1:30
SPEAKER_00: That you can just plug in and help out.
1:33
SPEAKER_00: Yeah.
1:33
SPEAKER_00: On call is never fun, but it's necessary.
1:36
SPEAKER_00: I have never had an issue with RubyGems ever.
1:38
SPEAKER_01: Yeah, I know.
1:39
SPEAKER_01: That's interesting, right?
1:40
SPEAKER_01: And I remember there being a lot of hand-wringing, especially a year ago, about well, what does this mean for the future of RubyGems?
1:46
SPEAKER_01: And I think as like an existential thing, I kind of understand it.
1:50
SPEAKER_01: I never really doubted that RubyGems itself would be up and secure and running.
1:55
SPEAKER_01: And I guess that's a tribute to all the past maintainers and all the present ones too, that we just don't, we are really lucky in the Ruby world.
2:00
SPEAKER_01: Like we just have not had, I'm not saying nothing has ever gone wrong.
2:04
SPEAKER_01: Of course, things go wrong.
2:05
SPEAKER_01: Right.
2:05
SPEAKER_01: That's a well-run chip.
2:06
SPEAKER_00: I'm not on like a platform role.
2:08
SPEAKER_00: I have been adjacent to them.
2:09
SPEAKER_00: And when you're building a bunch of machines and nodes all the time, and the Ruby gems has an outage, you do notice when the build fails for that reason.
2:17
SPEAKER_00: Yeah, of course.
2:18
SPEAKER_00: In that experience, it was very short-lived.
2:20
SPEAKER_00: Maybe an hour in the whole history that I've known.
2:22
SPEAKER_00: I think it's pretty good, V.
2:23
SPEAKER_00: I mean, you're impressive.
2:24
SPEAKER_01: If you're looking for better than that, your standards are prep probably a little too high.
2:27
SPEAKER_01: Yeah, that's good.
2:28
SPEAKER_00: Well, also, I hope you all the best that you don't get called.
2:31
SPEAKER_00: Yeah, I know.
2:32
SPEAKER_01: I hope I have the most boring job ever.
2:33
SPEAKER_01: But I'm glad to be I'm glad to be involved.
2:36
SPEAKER_00: Today, uh, you're lucky enough to listen in on our famous Between the Commits episodes, where we talk about all of the latest Ruby AI news briefings that we get regularly.
2:47
SPEAKER_00: And hopefully you can enjoy this roundup that we had.
2:49
SPEAKER_00: And again, if you have ideas and suggestions for what we should be talking about, feel free to let us know and we'll clue you in on how to do that at the end of the show.
2:58
SPEAKER_01: You could send it to news at therubyaipodcast.com, which may or may not be a valid email address, or you could just find us floating around the internet on various social media and suggest something.
3:08
SPEAKER_01: The cool thing is about this show that we have no idea what we're going to talk about until 10 minutes before the show airs, at which time we get a curated list of topics with some bullet points and links, and we just go.
3:19
SPEAKER_01: So you're getting our fresh and honest takes curated by the AI that we all know and love.
3:24
SPEAKER_01: Let's go.
3:25
SPEAKER_01: Yeah, let's do it.
3:26
SPEAKER_00: All right, so we may get interrupted, but maybe not from the timer.
3:29
SPEAKER_00: We'll find out.
3:30
SPEAKER_00: We'll find out.
3:30
SPEAKER_00: I pushed the button a little late.
3:31
SPEAKER_00: So you did push it?
3:32
SPEAKER_00: Because I don't see it.
3:33
SPEAKER_00: Do you see it?
3:34
SPEAKER_00: I did push it.
3:34
SPEAKER_00: Okay.
3:35
SPEAKER_00: I didn't push the video though.
3:36
SPEAKER_00: All right.
3:36
SPEAKER_00: There we go.
3:37
SPEAKER_00: Okay.
3:37
SPEAKER_00: Oh, there we go.
3:38
SPEAKER_00: Let's move it forward a bit.
3:39
SPEAKER_00: Oh, look at that.
3:40
SPEAKER_00: Aww snap.
3:41
SPEAKER_01: Oh, I didn't realize that you could also manipulate time.
3:44
SPEAKER_00: Okay.
3:44
SPEAKER_00: Hey, we're we're going.
3:45
SPEAKER_00: All right.
3:45
SPEAKER_00: All right, we're live.
3:46
unknown: Let's do it.
3:47
SPEAKER_00: The Rails team ran its own AI agents through a brutal new benchmark and caught one cheating.
3:53
SPEAKER_00: Uh-oh.
3:55
SPEAKER_00: So this is in a series of posts that Rubyon Rails.org has been posting on Agents on Rails.
4:02
SPEAKER_01: Uh-oh.
4:02
SPEAKER_01: Oh, that's the other audio.
4:04
SPEAKER_00: So the official Rails blog published results from the Agents on Reels stage three just yesterday, October 13th, testing whether coding agents can build on their own earlier work and not just ship one feature cold.
4:17
SPEAKER_00: Each task is a chain of tickets, a fresh session starts for each one, files and notes carry forward, but the conversation history doesn't.
4:25
SPEAKER_00: So closer to how a real team actually works.
4:28
SPEAKER_00: I'm pretty excited about this.
4:29
SPEAKER_00: I had built a similar thing for something I've been working on.
4:33
SPEAKER_00: Oh yeah.
4:33
SPEAKER_00: And I'm happy to see that this made it in.
4:35
SPEAKER_00: I was gonna upstream it and then I let my agent add this new link, and it was like, hey, it solved your problems.
4:41
SPEAKER_00: It's already done.
4:43
SPEAKER_00: And I was like, all right, fantastic.
4:45
SPEAKER_00: Let me let me plug into that, you know.
4:46
SPEAKER_00: Yeah.
4:46
SPEAKER_00: So I'm really excited to dig into this in more detail.
4:49
SPEAKER_00: I haven't.
4:50
SPEAKER_00: Definitely interesting to see this come to fruition.
4:53
SPEAKER_00: Are you doing any uh maybe benchmarks for agent harnessing like this?
4:59
SPEAKER_00: What do you think of this?
5:00
SPEAKER_01: In my personal work, I am not, and we're not really called upon to do it our service work today, although, which is to say, no, we do a lot of what other teams do, which is a lot of like introspection and talking with each other.
5:13
SPEAKER_01: We don't have a solid benchmark.
5:14
SPEAKER_01: We'll lean on people that do.
5:16
SPEAKER_01: I think that this is interesting for a couple of reasons.
5:19
SPEAKER_01: And I think the first is that is it really cheating?
5:21
SPEAKER_01: I mean, Deep Seek searched its own prior Git history mid-task to find a solution, and it found its own earlier attempt, so of course it didn't work.
5:29
SPEAKER_01: I'm curious to know like, can you give us a little bit more of the context on what it means that a fresh session starts and the conversation history does not carry forward?
5:40
SPEAKER_00: So, how I took it is the tasks are sequenced, like external documents.
5:47
SPEAKER_00: They're calling them tickets.
5:48
SPEAKER_00: So they have a series of tasks that they want a specific goal to get accomplished, and for each session, they get files and notes that were generated from the previous task carried forward.
5:58
SPEAKER_00: The actual conversation history that happened from those previous sessions does not.
6:02
SPEAKER_00: Okay.
6:02
SPEAKER_00: The idea being that every new ticket starts with a fresh clawed session based on what was produced from another previous session.
6:11
SPEAKER_02: Okay.
6:11
SPEAKER_00: If you're a human working through a bunch of tasks, you have a bunch of tickets that maybe needed sequencing, and you only know the state of the code as it evolves.
6:19
SPEAKER_00: Sure.
6:19
SPEAKER_00: I think it's trying to kind of go with that.
6:22
SPEAKER_01: What's the goal?
6:23
SPEAKER_01: I mean, what if it does this really well?
6:25
SPEAKER_01: Is it that you could then be more confident in a team of agents rather than single threading?
6:29
SPEAKER_00: I think the goal is more model capacity.
6:32
SPEAKER_00: Given a state, can one model do a better job than another model?
6:36
SPEAKER_00: I see.
6:37
SPEAKER_00: And so each task accomplishment is a different state.
6:40
SPEAKER_00: Okay.
6:40
SPEAKER_00: Even though they are sequenced and you would expect the sequencing to happen, can a model achieve does it achieve better performance throughout the tasks, or does it achieve better performance across a series of tasks?
6:52
SPEAKER_00: Now, this kind of gives both of those measurements a place that you can then check and benchmark against different models for both cases.
7:00
SPEAKER_00: Yeah, again, like deep seek searching its history for a solution, is that from a model perspective, yes.
7:08
SPEAKER_01: Right?
7:09
SPEAKER_01: I always go back to that means like agents aren't trying to cheat.
7:12
SPEAKER_01: It means that the guardrails weren't put in place specifically enough by the humans, right?
7:17
SPEAKER_01: It's a little bit like, well, we're trying to blame a machine, but really we just didn't write things specifically enough, and an agent found a way around it, I think.
7:25
SPEAKER_01: I'm also interested in this list of models that were the best.
7:29
SPEAKER_01: You know, Opus 5.5 was the most accurate, GPT-6 was the cheapest, and then GPT-6 is kind of all over this.
7:37
SPEAKER_01: They seem like they are the kind of winners at this point.
7:40
SPEAKER_01: I didn't see anything about Fable.
7:42
SPEAKER_01: Does that strike you as odd?
7:43
SPEAKER_01: It does.
7:44
SPEAKER_01: I don't know if that's just because they didn't do it.
7:46
SPEAKER_01: They didn't use it.
7:47
SPEAKER_01: Nobody can afford Fable anymore.
7:49
SPEAKER_00: I don't see Fable on this page, even though it's like, nah, you know, Opus 5.5 is said to be better than Fable, so who cares?
7:57
SPEAKER_00: Yeah.
7:58
unknown: Yeah.
7:59
SPEAKER_00: Good enough.
8:00
SPEAKER_00: Yeah, I'm excited.
8:01
SPEAKER_00: I know we got a switch, but I am excited to see maybe how this might evolve into harness evals, which I haven't seen yet.
8:08
SPEAKER_00: I am with you on that, and I'd like to see some of those.
8:10
SPEAKER_01: All right, let's get into this.
8:11
SPEAKER_01: So, headline a longtime Rails maintainer wrote the clearest rebuttal yet to DHH's Rust keynote, which I guess is what we'll call it now.
8:19
SPEAKER_01: Ernesto Tagworker, friend of the show, saw him, got to hang out with him a little bit at Rails World.
8:23
SPEAKER_01: So he wrote a little bit of a response here to DHH's Rails World keynote.
8:28
SPEAKER_01: Now, I've written a response, and Andy Kroll's written a response, and Ernesto's written a response, and there's like a hundred other people that have written a response.
8:35
SPEAKER_01: And so let's just make this the last week that we have to talk about DHH responses.
8:40
SPEAKER_01: That aside, this was actually quite good because Ernesto, as you know, V, he's got Ambu Labs and he also has FastRuby.
8:46
SPEAKER_01: FastRuby is a Rails upgrade agency.
8:49
SPEAKER_01: So this is what he does.
8:51
SPEAKER_01: His objection to the hey, let's just write it in Rust argument that DHH put forth is that just being more performant doesn't justify the maintenance cost of a second language in the stack.
9:03
SPEAKER_01: And so the question I guess we're asking here is is someone still has to understand the code a real constraint for something like a Rust rewrite or any kind of rewrite?
9:12
SPEAKER_01: Or is it just what every maintainer says and then, or is it just something we're telling ourselves now, I guess, and then six months from now we're all rewriting all of our apps?
9:20
SPEAKER_00: I have very specific experience with this, which is funny.
9:23
SPEAKER_00: At my previous job at Gusto, there was a lot of TypeScript evolving with all of the agents that they have a huge GraphQL presence at the company.
9:31
SPEAKER_00: Right.
9:31
SPEAKER_00: TypeScript is great marriage for a typed back end.
9:35
SPEAKER_00: So there are a lot of advantages with going with TypeScript.
9:37
SPEAKER_00: But when you don't have people with the expertise of TypeScript, because of the typing aspect, what you get is leaning on the agents for approval and quality, and is it doing the right thing?
9:49
SPEAKER_00: And is shaping the types in the right ways, right?
9:52
SPEAKER_00: Like there's a lot of language-specific things, even in something like Rust, it makes some affordances and optimizations based on the task you're giving it.
9:59
SPEAKER_00: Yeah.
9:59
SPEAKER_00: And based on the thing that you're working on in the immediate moment that don't become clear that you're making the wrong optimization up front until later.
10:06
SPEAKER_01: Yeah.
10:06
SPEAKER_00: If you're familiar with Rust, you would see that right away as you're reviewing, if you're reviewing.
10:11
SPEAKER_01: Yeah.
10:11
SPEAKER_01: If you're reviewing, yeah.
10:12
SPEAKER_00: So with an agent, because you have that context and because you have those forethoughts in order to instruct it the right ways for that language, you can get those optimizations more frequently up front as you're evolving that system that you're building.
10:26
SPEAKER_02: Yeah.
10:26
SPEAKER_00: I guarantee that that is also happening with whatever DHH may be producing without having a backlog of experience in Rust.
10:34
SPEAKER_02: Yeah.
10:34
SPEAKER_00: I can guarantee it.
10:35
SPEAKER_00: Yeah.
10:36
SPEAKER_00: And anybody for that matter, regardless of how many years of experience you have, if you have zero experience in another thing, it's going to show.
10:42
SPEAKER_00: And while the systems may be put together in a good way from a maybe abstract sense, those more granular things will catch up.
10:50
SPEAKER_01: Yeah.
10:51
SPEAKER_01: I really want to disagree with you, and I really want to disagree with Ernesto, but I just can't.
10:55
SPEAKER_01: In my gut, I want to react to say, look, these agents are improving.
11:00
SPEAKER_01: Look how far we've come in the last 10, 11 months, really, because everybody's marking November something of 2025 as like the day that everything changed for agentic development.
11:10
SPEAKER_01: So I want to say this objection is short-lived, and we'll all be able to just write whatever we want in whatever language we want soon enough.
11:17
SPEAKER_01: There's something in me that just sees too much evidence to the contrary right now to agree to that.
11:21
SPEAKER_01: I think your objection here is totally spot on.
11:24
SPEAKER_01: I think that Ernesto's is as well.
11:27
SPEAKER_01: Now, I can't say anything about how many hundreds of applications he may have upgraded.
11:31
SPEAKER_01: Probably some of them are candidates to be written in another language.
11:34
SPEAKER_01: Why not?
11:34
SPEAKER_01: People pick the wrong language all the time.
11:36
SPEAKER_01: Applications evolve all the time to need different technologies.
11:39
SPEAKER_01: But if you're just looking at a one-to-one rewrite, which is the example DHH gave us, you know, at 37 Signals, which is rewriting hey in Rust, there's a lot to be desired.
11:49
SPEAKER_01: Rewriting with agents leaves a lot to be desired in the experience category.
11:53
SPEAKER_01: And I was just interviewing somebody yesterday who was telling me a very similar story about rewriting a Rails application.
11:59
SPEAKER_01: They were using Go.
12:01
SPEAKER_01: They did a lot of agentic development.
12:03
SPEAKER_01: And this candidate was like, nothing wrong with Go.
12:06
SPEAKER_01: I like Go.
12:07
SPEAKER_01: What happened is that they ended up using Go routines everywhere, and nobody really knew how and when to use the Go routines.
12:14
SPEAKER_01: And that was a real problem, you know, a real problem in production that then needed to be cleaned up and solved.
12:19
SPEAKER_01: Right.
12:19
SPEAKER_01: After, you know, they let go of like half the company.
12:22
SPEAKER_01: That's just the world we live in today, right?
12:24
SPEAKER_00: The bell of the ball.
12:28
SPEAKER_00: Yeah.
12:28
SPEAKER_00: One final note on this.
12:30
SPEAKER_00: I feel like a lot is getting lost on benchmarks.
12:33
SPEAKER_00: DH maybe made a mistake trying to claim benchmarks as a proof that all of this is working.
12:40
SPEAKER_00: Because there's so many basis claims being made in the benchmark.
12:43
SPEAKER_00: It's a great point.
12:44
SPEAKER_00: Maybe a lot is getting lost in his point.
12:46
SPEAKER_00: I think I'm not going to speak for him.
12:48
SPEAKER_00: I see a point of an outcome of it being just you can shape an agent to perform a task for you and make something more performant in a very specific way.
12:56
SPEAKER_00: Sure.
12:57
SPEAKER_00: And basically build in languages that you're not familiar with as an outcome, if you desire that, nothing's going to stop you and you will get an outcome.
13:03
SPEAKER_00: Yeah.
13:04
SPEAKER_00: We're becoming less and less of that being an issue where that matters.
13:07
SPEAKER_00: Yeah.
13:08
SPEAKER_00: And that was a good takeaway that I got from that.
13:10
SPEAKER_00: But yeah, I think that's fair.
13:11
SPEAKER_00: But, you know, don't don't make claims.
13:15
SPEAKER_00: That had no backache.
13:16
SPEAKER_00: Uh that's another good thing to take away from the time.
13:21
SPEAKER_00: Next up here we have Ruby LLM shipped a feature that lets your code grade an AI feature before it ships.
13:28
SPEAKER_00: Super awesome.
13:29
SPEAKER_00: Yeah.
13:29
SPEAKER_00: Ruby LLM 2.1 landed with judges, a typed question about your own data, a yes-no probability, a choice among options, a score against a rubric that returned an actual measured probability, not a confidence number a model wrote as text.
13:44
SPEAKER_00: This is leaning into all of these system one models that have been coming out, like Jeb and Leia and Yeah.
13:50
SPEAKER_01: And could you, before we get into the rest of this, can you help educate me a little bit?
13:54
SPEAKER_01: Because here it says that it's an actual measured probability, not a confidence number.
13:58
SPEAKER_01: Is Jeb's typically a confidence number that you're getting?
14:01
SPEAKER_01: Or is it a measured probability?
14:03
SPEAKER_01: It's a measured probability.
14:04
SPEAKER_01: It is okay.
14:04
SPEAKER_00: In the same way that LLMs pseudo-deterministically generate the next likely word, it is that probability, a similar probability, except without text chaining, it's more structured data chaining.
14:18
SPEAKER_00: You give it some options up front, and those options from a data perspective are that next token of outcome.
14:26
SPEAKER_00: That's how the models work.
14:27
SPEAKER_00: By giving it these structured components up front, you can make those same pathway jumps that LLMs make just using a different undercarriage.
14:35
SPEAKER_00: Okay.
14:36
SPEAKER_00: They're not perfect, they are still probabilities, but they when you give them the choices upfront, you can if it's above a certain percent probability, those particular cases, you can be very confident in it.
14:47
SPEAKER_02: Okay.
14:48
SPEAKER_00: What ends up happening is when you build with these kinds of systems, the judgments that are very clear are very likely to be true.
14:55
SPEAKER_00: And so what you end up doing is building off of those cases and then dropping down to something else to handle the ones where it's less a little bit confused.
15:05
SPEAKER_01: Oh, okay.
15:06
SPEAKER_01: I'm starting to see this, I think, a little bit clearly now.
15:09
SPEAKER_01: So I can be running a system, an AI, you know, an agentic system, or just any kind of AI feature, and I can evaluate the data that I'm getting.
15:15
SPEAKER_01: And so if I feel like the confidence that we've responded in a certain way is very highly likely to be accurate, then I can kind of ignore those, but then spend more time or more resources on the ones where we're less confident.
15:26
SPEAKER_00: Exactly.
15:27
SPEAKER_00: It helps you build the systems a little bit differently, right?
15:29
SPEAKER_00: Where you have a necessity for confidence, you can skim off the top and let AI handle those cases.
15:36
SPEAKER_00: So this is kind of like a way of doing that.
15:37
SPEAKER_00: So really interesting to see letting features ship themselves.
15:41
SPEAKER_00: Yeah, right.
15:42
SPEAKER_00: I'm excited to announce hopefully by the time this comes out, the feature machine gem, which will pick up on a lot of this stuff.
15:48
SPEAKER_00: Oh, let's hear it.
15:49
SPEAKER_00: It's a Rails gem that lets you mount a feature machine so that you end users can request features of your application or report bugs.
15:58
SPEAKER_00: Oh, yeah, I love this.
16:00
SPEAKER_00: You can attach your local agents to the feature machine MCP and pick off stuff.
16:06
SPEAKER_00: Oh, I like this.
16:07
SPEAKER_00: And then automate the feedback loop of your end users and let the users build the machine.
16:12
SPEAKER_01: I read about this concept in a Steve Yeagay post uh about a month or two ago.
16:17
SPEAKER_01: He's doing it on his game, which I I don't really know much about.
16:20
SPEAKER_01: It sounds like a really excellent direction to go in for just customer satisfaction, customer success for a lot of different applications.
16:27
SPEAKER_01: I mean, it doesn't have to be SaaS, it could be anything, right?
16:29
SPEAKER_01: But it's like, oh man, I really wish this thing would do X.
16:31
SPEAKER_01: Now maybe it can just do that, right?
16:33
SPEAKER_01: Just ask them.
16:34
SPEAKER_01: You know, it's amazing.
16:36
SPEAKER_00: Obviously, a lot of uh edge cases here.
16:38
SPEAKER_00: Sure.
16:39
SPEAKER_00: You know, it kind of requires somebody to approve the features and bugs that come in so that people aren't just building anything.
16:46
SPEAKER_00: Yeah, I'm excited to announce it.
16:47
SPEAKER_00: Check out Ruby LLM's judges, they are fantastic and they work great.
16:51
SPEAKER_00: Yeah.
16:52
SPEAKER_00: Hopefully we can replace your whole eval pipeline with it.
16:54
SPEAKER_00: This is another reminder that we need to get Carmine back on the show.
16:56
SPEAKER_01: Yeah, totally.
16:57
SPEAKER_01: Because it seems like he's built an entire world since the last time we talked to him about this.
17:02
SPEAKER_01: Next, Obi Fernandez, friend of the show and leader of Czar on the tech side, has a one-gem answer to which AI decision model do I use?
17:11
SPEAKER_01: And it is Ruby decision model, which just hit 0.3.0.
17:16
SPEAKER_01: So this is a single Ruby client that talks to every major quote unquote decision model.
17:20
SPEAKER_01: So JEV, like we were just talking about, for example, and we're gonna talk about this later in the show.
17:25
SPEAKER_01: Everybody's got their own JEV now.
17:26
SPEAKER_01: A Ruby decision model can help you to pick the right one.
17:29
SPEAKER_01: It's a transport layer for Obi's broader quote unquote impatient programming push.
17:34
SPEAKER_01: And so that is actually what I got out of this, which is that there's a revival of extreme programming uh or extreme programming's rules, and it's called impatient programming.
17:42
SPEAKER_01: So, V, as the closest person to probably Ruby decision model slash Obi slash impatient programming, where do you want to start?
17:51
SPEAKER_00: This is fantastic.
17:52
SPEAKER_00: Yeah.
17:53
SPEAKER_00: Ruby decision model, it can back the Ruby LM Judge as an example, if you would like it to, where you basically set up a bunch of decisions that need to get made or questions you want answered, and you set those up front, it scores them and tells you probabilities of those answers for whatever you give it.
18:10
SPEAKER_00: Really cool, really interesting.
18:12
SPEAKER_00: And the biggest part, like you're mentioning, is you can give it any back end you want for that provider.
18:17
SPEAKER_00: So whether it's open router that you want to hook into any number of model decision models, it supports Clef now, which is an Omni model.
18:25
SPEAKER_00: You can give it audio and video.
18:27
SPEAKER_00: Oh, interesting.
18:28
SPEAKER_00: And let it decide on those different types of modes.
18:32
SPEAKER_00: I love the interface of it.
18:33
SPEAKER_00: It's been working really well for me personally.
18:35
SPEAKER_00: Oh, really?
18:36
SPEAKER_00: These models are, you know, Clef is probably the best one that I've tried personally as well.
18:40
SPEAKER_00: And it does the confidence scores once you get them high enough, the outcomes you can trust empirically.
18:46
SPEAKER_00: Yeah, I've been really loving it.
18:47
SPEAKER_01: Can you walk me through this a little bit?
18:48
SPEAKER_01: So, Ruby Decision Model is being used on the back end, it is helping you to make a decision.
18:53
SPEAKER_01: Is it deciding in real time which model to use based on some input, or is it something that is helping you evolve or sort of iterate on the models that you use for different situations in your evaluation stack?
19:03
SPEAKER_00: So it's not a model decider.
19:04
SPEAKER_00: I mean, you could build a model decider out of it.
19:07
SPEAKER_00: Oh, oh.
19:08
SPEAKER_00: Maybe I'm misunderstanding this then.
19:09
SPEAKER_00: Okay.
19:10
SPEAKER_00: This is basically an interface to the system one models.
19:13
SPEAKER_00: Ah, okay.
19:14
SPEAKER_00: Okay, got it.
19:15
SPEAKER_00: So, like similar to the Ruby LLM Judge, this is like a wrapper around making those specific decisions.
19:21
SPEAKER_00: So instead of judgments, it's more of a QA shape.
19:23
SPEAKER_00: Okay.
19:24
SPEAKER_00: So that you can get legit answers to a series of questions that you want out of it.
19:28
SPEAKER_01: Okay, thank you.
19:28
SPEAKER_01: Very similar to Ruby LLM Judge.
19:30
SPEAKER_01: I guess if you don't want or need the entire Ruby LLM feature set, you could use just this one.
19:35
SPEAKER_00: Yep, exactly.
19:36
SPEAKER_01: What about inpatient programming?
19:38
SPEAKER_01: So I just signed up, by the way.
19:39
SPEAKER_01: I am number 301.
19:41
SPEAKER_01: So you can go to this inpatient programming website, and it looks exactly like a website from 1999 in honor of extreme programming.
19:48
SPEAKER_01: It's kind of awesome.
19:49
SPEAKER_01: And it's got these five different rules, but there's not a lot that's actually been written about it yet.
19:54
SPEAKER_01: This is kind of like this is a placeholder page.
19:56
SPEAKER_01: So what's to come?
19:57
SPEAKER_00: This is gonna be huge.
19:58
SPEAKER_00: And so it's bringing back all of those core principles from extreme programming, bringing forward the entire realm of how software of quality kind of like got boring.
20:08
SPEAKER_00: There's a bunch of people contributing to it behind the scenes.
20:11
SPEAKER_00: Okay.
20:11
SPEAKER_00: So that as an example, people are writing basically articles on how a particular facet of designing quality software in a very particular way has impacted them and how they make use of it.
20:22
SPEAKER_00: What is kind of evolving out of it is kind of that material of how to apply the principles outlined in kind of like any number of ways.
20:30
SPEAKER_00: Yeah.
20:31
SPEAKER_00: There's a tool set coming out to help support building on top of all these principles and ways of working that you can follow.
20:37
SPEAKER_00: I think there's a GitHub page for it.
20:39
SPEAKER_00: Oh, there is?
20:40
SPEAKER_00: Okay.
20:40
SPEAKER_00: I'll check that out too.
20:41
SPEAKER_00: It's called Tools for Imps.
20:44
unknown: All right.
20:44
SPEAKER_00: And you'll see there's a RubyGem Exhale Ruby, which a contraction gate for Rails now in part of this RubyGem that helps you reduce duplication and a number of other principles from all right, you've generated all this stuff now, contract and bring it back in.
20:59
SPEAKER_00: Bring back these principles of cycling and refactoring into your changes that you make, right?
21:04
SPEAKER_00: On a regular capacity.
21:05
SPEAKER_00: Yeah.
21:05
SPEAKER_00: Where you're you're kind of doing this breath work of exhaling all of the output and then reeling it back in and honing in on what changes you're actually making to the system over time.
21:15
SPEAKER_01: That's just like one aspect of it.
21:16
SPEAKER_01: I'm fascinated by this.
21:17
SPEAKER_01: I know we have to move on to the next topic, but I'd like to talk more about this on a show to come.
21:21
SPEAKER_01: You know, I've been writing about new principles for software engineers to adhere to in this kind of next generation of software development.
21:28
SPEAKER_01: And I just think that we need more people that are out there thinking and writing and building tools for exactly this moment.
21:33
SPEAKER_01: So I'm excited about it.
21:35
SPEAKER_00: Yeah, there's a lot to come.
21:37
SPEAKER_00: You know, there's a bunch of rules kind of outlined, values.
21:40
SPEAKER_00: Rules.
21:40
SPEAKER_00: I love rules.
21:41
SPEAKER_00: Oh, yeah.
21:42
SPEAKER_00: Values too.
21:42
SPEAKER_00: Case studies, right?
21:43
SPEAKER_00: It's pretty in-depth.
21:44
SPEAKER_00: Okay.
21:45
SPEAKER_00: It signed up because this is gonna be a huge change in software in this new age for sure.
21:51
SPEAKER_00: All right.
21:51
SPEAKER_00: A lot of great stuff coming out.
21:53
SPEAKER_01: All right, I'm signed up.
21:53
SPEAKER_01: I'm gonna check it out.
21:55
SPEAKER_01: All right, Google's new work agent can hire itself a coworker with its own inbox and Calendar.
22:01
SPEAKER_01: I just had to laugh at this.
22:03
SPEAKER_01: So Google Cloud unveiled the Gemini Agent on October 8th, so six days ago as of this recording.
22:09
SPEAKER_01: And so it answers questions, it works on assigned objectives for hours or days.
22:13
SPEAKER_01: It runs scheduled tasks, it writes code.
22:15
SPEAKER_01: Google basically was, hey, Meta just came out with a thing and it's doing really well.
22:21
SPEAKER_01: How can we get that plus get a bunch of people to buy more licenses for a Gmail and calendar?
22:27
SPEAKER_01: So it's in private preview right now.
22:30
SPEAKER_01: So I have not been able to check it out yet.
22:32
SPEAKER_01: Or maybe I can, but I haven't.
22:34
SPEAKER_01: It's got a project level spend cap, it's got choice for models.
22:37
SPEAKER_01: So you can use Gemini, but today you can also use Claude and more models to come.
22:42
SPEAKER_01: Okay.
22:42
SPEAKER_01: You probably already can tell that I'm a little bit disenchanted by this item.
22:46
SPEAKER_01: But Valentino, what do you think about a coworker agent sending email in its own name and actually accountable for what it said?
22:53
SPEAKER_01: Sorry, who's actually accountable for what it said?
22:56
SPEAKER_01: The agent, the person who configured it, or no one.
22:58
SPEAKER_01: And actually, this is happening all the time at Czar.
23:00
SPEAKER_01: So what do you think?
23:01
SPEAKER_00: As a human, you're accountable.
23:02
SPEAKER_00: If you control these things, you're accountable.
23:05
SPEAKER_01: I think that's true.
23:06
SPEAKER_01: Although, in some cases, I think you're really probably more accountable to your reaction to its responses or its answers, right?
23:12
SPEAKER_01: You know, an agent can come on and say, Oh, hey, I saw this thing, or I'm about to go and do this thing, and you still have to like either give it permission or take some action based on the information that you get.
23:22
SPEAKER_00: I've got a great example.
23:23
SPEAKER_00: All right.
23:23
SPEAKER_00: You get a bot doing these things, wanted to email people.
23:26
SPEAKER_00: First couple go out, great.
23:28
SPEAKER_00: You get a wildcard email.
23:29
SPEAKER_00: It just says things it shouldn't or discloses stuff it shouldn't, and you turn it off because the value isn't there.
23:35
SPEAKER_00: I'm responsible for it emailing.
23:37
SPEAKER_00: And if it can't email appropriately, then I'm not going to give it that permission.
23:41
SPEAKER_00: Right.
23:42
SPEAKER_00: It's all about guardrails.
23:43
SPEAKER_00: If you put the right guardrails in place and it works for you, then who cares?
23:47
SPEAKER_00: Yeah, it's a fair point.
23:48
SPEAKER_00: You'll get some one-offs and you correct them and you apologize.
23:51
SPEAKER_00: You take onus for that the mistake.
23:54
SPEAKER_00: We all make make mistakes.
23:55
SPEAKER_00: Yeah.
23:55
SPEAKER_00: Whether or not it's an agent or you personally that is making the mistake, I don't think matters at this point, as long as you take responsibility for that action and outcome.
24:04
SPEAKER_01: I think it's a fair point, but I also think that personal responsibility is not something that necessarily comes about on its own in the enterprise.
24:11
SPEAKER_01: Let's face it, that's where this is being sold.
24:13
SPEAKER_01: A company like mine is not buying a thing to read my email and send my email.
24:16
SPEAKER_01: And if I wanted something to do that, I would build it and I would make it probably better than the off-the-shelf thing.
24:21
SPEAKER_01: But I can see, you know, a bunch of enterprises that are using Google saying, oh, well, wouldn't this be great?
24:25
SPEAKER_01: Let's spin up a couple hundred of them and then they run rampant because people aren't putting guardrails in place.
24:30
SPEAKER_01: And then they need somebody to blame, only they can't blame Google because they've probably put it in their terms of service that you can't.
24:36
SPEAKER_01: So you gotta blame somebody else.
24:37
SPEAKER_00: Here's an example of where a guardrail has worked out well, and specifically with email.
24:42
SPEAKER_00: Okay.
24:42
SPEAKER_00: ChatGPT lets you connect Gmail as an example.
24:45
SPEAKER_00: Just by you connecting to it doesn't mean it's gonna reference it all the time.
24:49
SPEAKER_00: It doesn't reference it unless you explicitly ask it to.
24:51
SPEAKER_00: So I have to say, hey, check my email for this, right?
24:55
SPEAKER_00: Yep.
24:55
SPEAKER_00: I want to send an email to this person and it asks me, do you want to send this in a draft form?
25:01
SPEAKER_00: Right?
25:01
SPEAKER_00: Those guardrails are in place to not automate it.
25:04
SPEAKER_00: Now I can set up a loop and I can say, hey, check for any email from this person and let me know about it on a schedule.
25:10
SPEAKER_02: Yeah.
25:11
SPEAKER_00: Then I'll get notice that, hey, I got an email.
25:13
SPEAKER_00: Do you want to respond about it?
25:15
SPEAKER_00: And I can set up another loop that then automates that.
25:17
SPEAKER_00: Then I'm responsible for that automation that I set up, right?
25:20
SPEAKER_00: But at least the guard rules are there preventing it from happening right off the bat.
25:24
SPEAKER_00: Yeah, it's a fair point.
25:24
SPEAKER_00: You can have all of it, is what I'm saying.
25:27
SPEAKER_00: You can have it all.
25:29
SPEAKER_00: Next up, we've got Anthropic just dropped its blanket ban on AI personalized political outreach.
25:36
SPEAKER_00: Anthropic, known for its policy, recently updated its usage policy, published October 8th and effective November 12th, removes the flat prohibition on personalized voter and campaign targeting.
25:47
SPEAKER_00: They say that the old blanket ban was blocking legitimate civic work like nonprofits translating voter information or election officials sending ballot correction notices.
25:58
SPEAKER_00: Deception is still banned outright.
26:00
SPEAKER_00: So false information.
26:01
SPEAKER_00: Thank God.
26:02
SPEAKER_00: Thank God we can't be deceived by these AIs.
26:05
SPEAKER_00: False information about candidates are voting, impersonating officials, voter suppression, and hiding who's actually behind a message all stay prohibited under a renamed narrower section.
26:17
SPEAKER_01: So I love this question that was posed for us by AI.
26:20
SPEAKER_01: So can a policy really hold the line between a nonprofit personalizing voter information and a campaign micro-targeting you with AI?
26:27
SPEAKER_01: I mean, don't we already know that the answer is emphatically no?
26:30
SPEAKER_01: Right.
26:30
SPEAKER_01: This wasn't even possible in the social media days when there was no generative AI because nobody could figure out what was or wasn't fraudulent or just flat out lying or deceptive.
26:40
SPEAKER_01: And everybody's opinion of it is different.
26:43
SPEAKER_01: Talk about non-deterministic, I just think when these things come out, it really is just more posturing.
26:49
SPEAKER_01: Anthropic wants to be the more responsible one.
26:51
SPEAKER_01: They're the ones that are not going to work with the DOD, right?
26:53
SPEAKER_01: And ChatGPT is so they could be the evil ones.
26:56
SPEAKER_01: And so, you know, you see more of this that comes out.
26:58
SPEAKER_01: And I'm not saying that Anthropic is doing anything necessarily untoward, and maybe there is a kernel of change that comes with this policy.
27:05
SPEAKER_01: But I think the policy itself is really just marketing.
27:08
SPEAKER_00: Yeah, I don't disagree.
27:09
SPEAKER_00: I think people are too creative.
27:12
SPEAKER_00: And the craziest example I've seen is somebody built a prompt injection attack vector through a site's XML.
27:20
SPEAKER_00: Oh, like bots go and they read your site from uh, hey, what's on this page?
27:25
SPEAKER_00: They built a prompt injection attack vector by the navigation so that when it visited the page, it got, oh, okay, I should look at the site map in that site map.
27:36
SPEAKER_00: It then let the thing observing it so an agent have the ability to create any page that they want with any parameters that they want, uh-huh, and then basically let them build and then fill out a form of personal information using one letter at a time because they would have to spell out the index on the site map and type the message one page at a time.
27:57
SPEAKER_00: Oh my god.
27:58
SPEAKER_00: And so they're able to exfiltrate data basically by site navigation.
28:02
SPEAKER_00: That is creative.
28:03
SPEAKER_00: One example of many, right?
28:05
SPEAKER_00: And it's endless.
28:06
SPEAKER_00: And now that agents are involved, I can't imagine what creativity they have.
28:09
SPEAKER_01: Yeah, it really is true.
28:10
SPEAKER_01: And we know that if you just set it on a problem, this was Mike Delesio last week, right?
28:14
SPEAKER_01: Right.
28:14
SPEAKER_01: You tell the agent to do it, it comes back with some maybe half-assed solution, and then you say, Hey, is that the best you can do?
28:21
SPEAKER_01: Right.
28:21
SPEAKER_01: And it goes on and say, No, actually, you can do plenty of a lot more damage if you really give me some time uh and some tokens.
28:28
SPEAKER_00: Yeah, nothing a policy is gonna fix.
28:30
SPEAKER_00: Yeah.
28:31
SPEAKER_01: Uh yeah, yeah, it's absolutely true.
28:33
SPEAKER_00: Our final question of the day here is based around Microsoft's launching of a model that doesn't write text, it scores probability.
28:40
SPEAKER_00: Hey, I see a theme forming here.
28:43
SPEAKER_00: Microsoft's decisional one model, they went live on October 9th in Microsoft Foundry.
28:48
SPEAKER_00: Uh so instead of generating a sponsor, it returns a calibrated probability against a fixed set of options you define.
28:55
SPEAKER_00: Yes, no, multiple choice or rubric score.
28:58
SPEAKER_00: It was built on post-trained Quen 3.5, which is interesting.
29:03
SPEAKER_00: Yes, that is.
29:03
SPEAKER_00: And priced at just over four cents.
29:06
SPEAKER_00: So low, low price.
29:07
SPEAKER_00: Low, low price of four cents a million input tokens and output being free, which is maybe the cheapest one we've seen.
29:13
SPEAKER_00: Microsoft says it'll rebase onto its own OpenEI model soon, which is interesting.
29:18
SPEAKER_00: The question here is with Microsoft Cloudflare and startup shipping compatible decision models in the same month, is this a real new category or a feature that every major vendor was always going to copy within weeks?
29:32
SPEAKER_01: Good question, actually.
29:33
SPEAKER_01: And I think that every major vendor was always going to copy it within weeks.
29:36
SPEAKER_01: And this is from the JEV announcement, right, from Type Sense.
29:39
SPEAKER_01: And it was a big deal, it caused a big wave.
29:41
SPEAKER_01: You know, I see that as a line in the sand to some degree or kind of staking your territory.
29:46
SPEAKER_01: And then, especially in the age of AI, you've got to fight really, really hard to defend it.
29:50
SPEAKER_01: That said, this seems a lot bigger than a feature to me.
29:52
SPEAKER_01: And I think this show has maybe even convinced me of that more that people are genuinely very excited for these models and what they can do and the possibilities that they open.
30:02
SPEAKER_01: And I think, Valentino, you've already given us a couple of good examples of how these things are used in the real world.
30:07
SPEAKER_01: And I think we're really just starting to scratch the surface.
30:09
SPEAKER_01: I mean, this announcement came out when I was on my way back from Rails World.
30:12
SPEAKER_01: So this is like not old news, right?
30:14
SPEAKER_01: So we're really just getting started with this.
30:16
SPEAKER_01: I think I know where you stand on this, but I'd love to hear your take as well.
30:18
SPEAKER_00: Decision trees and business processes have been a thing for decades.
30:22
SPEAKER_00: Yeah, it's a good point.
30:23
SPEAKER_00: This is basically just fulfilling that from an LLM perspective in a shaped type way.
30:27
SPEAKER_00: Yeah.
30:27
SPEAKER_00: I think it makes a lot of sense because of that reason, right?
30:30
SPEAKER_00: It fits right on top of that process tree already.
30:33
SPEAKER_00: And businesses have those shapes already defined.
30:36
SPEAKER_00: We have flows of users.
30:38
SPEAKER_00: This is how they matriculate into the system, this is how they use the system, this is how it applies to our business and outcomes for that.
30:44
SPEAKER_00: Businesses have all these shapes that it just like is literally a decision tree.
30:49
SPEAKER_00: It's beyond just customer service, which is what the chatbot was built for.
30:53
SPEAKER_02: Yeah.
30:53
SPEAKER_00: That even follows a decision tree.
30:55
SPEAKER_00: That's true.
30:55
SPEAKER_00: There's just so many things that have this need that got fulfilled, and it's obvious.
31:00
SPEAKER_00: I think the question should be were all of these companies using something like this internally and taking advantage of the fact that they had thought that they had an exclusive edge against the rest of the industry, and then it was exposed that somebody was doing this, right?
31:16
SPEAKER_00: And then, like, oh, I guess we'll package it up and release it.
31:20
SPEAKER_01: Either way, however, it got here.
31:23
SPEAKER_01: I'm glad it's here.
31:24
SPEAKER_01: And I think that it's fun to see these kinds of things make ripples in the industry.
31:29
SPEAKER_01: You see these news items come about, you see these kind of feature sets or brand new models come out, and then you get to see what people do with them.
31:37
SPEAKER_01: I wouldn't say that I am on the bleeding edge of that, but I love to watch the people that are on the bleeding edge, and then I'm kind of like the early adopter that comes along after that.
31:45
SPEAKER_01: I'm excited by what's here and what the potential is for us.
31:48
SPEAKER_01: And I think it's going to be a big part of how we shape our applications.
31:51
SPEAKER_00: New kinds of models are going to keep coming out.
31:53
SPEAKER_00: This one is great at decision trees.
31:55
SPEAKER_00: If you have a decision tree style problem that you need solved, look for one of these things, right?
32:00
SPEAKER_00: Yeah.
32:00
SPEAKER_00: Here is the solution.
32:01
SPEAKER_00: If there's very high probability of the outcomes of that, which you'll see right away, then you have a cheap and fast way, I think, is not being underlined enough here of solving that.
32:11
SPEAKER_00: That's a fair point.
32:11
SPEAKER_00: Yeah.
32:12
SPEAKER_00: Because these responses, they're like 10-15 milliseconds, some of them.
32:15
SPEAKER_00: Yeah.
32:15
SPEAKER_00: Getting that style of instant feedback on a decision from a model is just like wild when you play with it.
32:22
SPEAKER_00: All right.
32:22
SPEAKER_00: Well, I think we did enough damage for one day.
32:24
SPEAKER_00: This was a good one.
32:25
SPEAKER_00: Yeah.
32:25
SPEAKER_00: This is a good one.
32:26
SPEAKER_00: The age of decisions is what we've entered.
32:29
SPEAKER_00: Now we don't even need to decide on things.
32:31
SPEAKER_00: Thank God.
32:32
SPEAKER_01: I don't want to make any more decisions.
32:34
SPEAKER_01: I'll make the decision for all of you.
32:35
SPEAKER_01: Tune in again next week because we're going to have all new things to rant and rave about.
32:40
SPEAKER_01: And then coming up, either the next episode or the one after that, we have to look at the calendar.
32:45
SPEAKER_01: We've got Rick Van Michellen from the Museum of Modern Art.
32:48
SPEAKER_01: I think that's going to be really interesting to dive in and learn what's going on in the museum, which is a fully Ruby operation.
32:55
SPEAKER_01: Wow.
32:55
SPEAKER_01: You know, they do some real, obviously, like Moma.org does some real numbers and they're supporting it all on the back end with Ruby and a lot of AI.
33:02
SPEAKER_01: So we'll learn some more about that.
33:03
SPEAKER_00: Awesome.
33:03
SPEAKER_00: Yeah, I'm looking forward to that as well.
33:05
SPEAKER_01: All right, everyone.
33:05
SPEAKER_01: Thank you for joining us.
33:06
SPEAKER_01: Tune in again next week.
33:08
SPEAKER_01: We'll see you soon.
33:09
SPEAKER_00: See ya.