Latest / Startup Dad / He Trained a Trillion-Parameter Model on Dad Jokes for $200 | Adam Grenier (Dad of 3, AI Matey)

Transcript

He Trained a Trillion-Parameter Model on Dad Jokes for $200 | Adam Grenier (Dad of 3, AI Matey) · Startup Dad

• [00:00:00] Adam Grenier: first you learn all of the, hard math, and then you learn all of the data science, and then you learn all of the like, machine learning, and then you can start getting into model training, then you can get into reinforcement learning. [00:00:13] Adam Grenier: Or you can sit there with Claude and Onslaught and do it tonight, and you're gonna learn six things that most PhDs don't learn until they're like sixth year into this. [00:00:23] Adam Fishman: Welcome to Startup Dad, the podcast where we dive deep into the lives of dads who are also leaders in the world of startups and business. I'm your host, Adam Fishman. What do dad jokes and machine learning have in common? More than you might think, and today's guest can prove it. I'm joined by returning guest, Adam Grenier, growth and product advisor, creator of AI evaluation system, DadBench, and one of the most genuinely curious builders I know. [00:00:55] Adam Fishman: Adam is a dad of three, a high schooler, a middle schooler, and an eight-year-old, and he's back on the show three years after being one of our very first guests. Since we last spoke, Adam was laid off from Eventbrite and did what any reasonable person would do. He went 1,000% into AI experimentation. What started as an attempt to build an AI-first product studio became a deep hands-on education in how these models actually work from the inside out. [00:01:23] Adam Fishman: Today, we talked about how Adam taught himself reinforcement learning by training a model to write dad jokes on a hobby budget, what and Goodhart's law reveal about the surprising ways AI systems break when you're not careful, the unexpected parallels between improv comedy and model training, how to use AI as a genuine research partner without outsourcing your judgment entirely, and the very real and ongoing work of advocating for neurodivergent kids, including navigating two children with different diagnoses through the same school system at the same time. [00:01:57] Adam Fishman: If you like what you hear, please subscribe to Startup Dad on YouTube or Spotify so you never miss an episode. You'll find it everywhere you get your podcasts. I hope you enjoy today's conversation with Adam Grenier. [00:02:09] Adam Fishman: Welcome back, Adam Grenier, to Startup Dad. Adam, round two, three years in the making. We're here now. Thank you [00:02:19] Adam Grenier: You're welcome. This is exciting. This is exciting. Yeah, I, Did we record before you launched? [00:02:24] Adam Fishman: We did. We recorded like months before I launched 'cause I couldn't find anyone to edit the show, and I thought I would do it myself. Remember I was like playing sound effects? I was like, "Here's how Riverside works. Let me play a [00:02:37] Adam Grenier: You had a, you had one of those whistles for a while. You had [00:02:43] Adam Fishman: Oh man, what a, what a long, strange journey this has been. Yeah, so you, you were here on day one. I think you were actually the first recording I did, but the second show that I launched, which makes me think that the intro probably doesn't actually match the show 'cause I probably said, "My first ever guest," and you were my se- second. [00:03:03] Adam Grenier: who did you put ahead of me after [00:03:04] Adam Fishman: I think Nick Soman was, was [00:03:07] Adam Grenier: All right, I'll accept that. [00:03:09] Adam Fishman: it's, uh, it's okay. so June 2023, the show launches. You're one of the first voices. I've done now 150 plus episodes. I've, been nominated for two Webbys. I've won one. I was an honorable mention, so yeah, just dust that off. what do you remember about being a guinea pig for a brand new podcast, aside from the slide whistle and the, you know, cymbal noises and stuff? [00:03:35] Adam Grenier: Um, let's see. I think y- uh, you're shinier now. Um, so your lighting got better. Mine hasn't. I've tried in various different ways. it's a sunny day today, so it, it's messing up all of my shadows. I think, you know, especially for this podcast, right? Like, We didn't talk about anything I would normally talk about in a podcast, right? [00:03:56] Adam Grenier: So like, you didn't know if you'd ever have a listener or if anyone would care about that stuff. I think we were both maybe even a little, I don't know if timid's right. I don't think you or I ever really come across as timid, but the unsure of like how deep do I go into this kind of thing? [00:04:10] Adam Grenier: Are we gonna just totally lose people that don't care about this depth? should we be talking about cryptocurrency? Uh, [00:04:17] Adam Fishman: No. Heavens no. [00:04:22] Adam Fishman: Amazing. Amazing. Yeah, I didn't really know what thing, what would happen with this. So, a lot of what we spent talking about in the last show was about your son, Archie, who, is on the autism spectrum. I think Archie's [00:04:35] Adam Fishman: eight [00:04:35] Adam Fishman: now. He was like, he was a lot younger, well, obviously three years ago. [00:04:39] Adam Fishman: we talked a lot about advocacy, which, uh, I was very interested in. and you know, I brought that back up on the show with different guests. So I guess one of the things I want to do as a kind of like bookend is just check in and ask you how Archie's [00:04:51] Adam Fishman: doing what it looks like these days. [00:04:53] Adam Grenier: his given name is Archer, and he has evolved to that. So he decided between, uh, let's see, I think it was first and second grade that he would go by Archer now. I still call him Archie most of the time, and so does his mom, and like, but he'll get defensive about it now, that Archer is, the name. [00:05:08] Adam Grenier: Uh, he's doing great. you know, one of the things, especially with a lot of neurodiversity is, the younger you're helping and giving assistance to someone, the, the more, it just compounds really fast. Like, just like with language and things like that, like learning different skills at different ages, can be just completely different, [00:05:27] Adam Grenier: you know, different system. [00:05:28] Adam Grenier: Plus, you have all of the, like, preconceived ways that people treat you and do all of that kind of stuff, versus helping him build the tools and us build the tools. Like, when he was two is when, he was diagnosed, and we were able to kind of like start looking at that [00:05:41] Adam Grenier: an interesting challenge that we've had over the last couple years is that, our middle child is ADHD in like a very traditional sense of ADHD, and we had to fight to get her, support at school that she wasn't getting. So getting on like, um, [00:05:56] Adam Fishman: IEP, or something? [00:05:57] Adam Grenier: thank you. Uh, it, it's summer, so I forget all of the school acronyms. [00:06:01] Adam Grenier: I have one more week. an IEP, and meanwhile, he was asked to get off of one. And so we were, like, doing the opposite advocacy at the same time, or kind of push of like, wait a minute, like, he's very social and ver- like engages with people, but he struggles sometimes with, social cues and things like that. [00:06:17] Adam Grenier: And, you know, a boy in kindergarten and first grade, like they're just part of this like pack of boys. Like, they just get like consumed by it all. But, you know, as you get older, you start to need, you know, more one-on-one relationships and things like that. And so we, we had a lot of anxiety of like, well, is he gonna make that transition well if he doesn't have that support at school anymore? [00:06:38] Adam Grenier: If he's just kind of like treated like every student there, because he masks well. is he gonna actually like have a harder time in two years because now he lost that kind of one-on-one support that he was getting? And so he's technically, he's not on an IEP anymore. and our daughter is, but like both of them were kind of being told they don't need the thing. [00:06:56] Adam Grenier: so there was a lot of back and forth there. So just, that math and strategy and figuring out and dealing with school districts and everything like that, it's a never-ending journey, and it comes in lots of different flavors. [00:07:06] Adam Fishman: Yeah. And is your oldest daughter in high school now? Is that [00:07:10] Adam Grenier: Yeah. she'll be a sophomore this [00:07:11] Adam Fishman: Wow. That is wild that I've one more year until my daughter's in high school. So I've got an eighth grader and a sixth grader now, so, but what a time, what a time to be alive. Um, another thing I wanted to call back to is so three years ago we made, we were talking about Rivians. [00:07:28] Adam Fishman: I don't know if you recall. you've been two years waiting for one and, you know, I think our joke was that the company might go out of business before it actually shipped, which, [00:07:37] Adam Fishman: you know, still could happen. Yeah, back and forth. So did the Rivian ever show up or did you call it Audib- Oh, it did. [00:07:42] Adam Grenier: Yep, it it showed up. I no longer have it. So I, I had the Rivian for two years, and I loved it. I'm a big fan. it's a great vehicle. It's great for families, right? Like, it's not terribly great in the Oakland Hills with very narrow streets. [00:07:58] Adam Grenier: without going like, crazy, it's, it's one of the larger footprint six-passenger vehicles you can ask for. [00:08:06] Adam Grenier: and you know, it wasn't cheap. And so about, let's see, October of last year, I made the like had to let it go decision of like, I kinda hit the side of a car and felt like I was going to every single time I was driving in our neighborhood. and just from a financial perspective, it was like, "All right, this car payment's quite a bit." [00:08:23] Adam Grenier: Kids are a little older now. I could get something a bit smaller footprint that makes sense in our neighborhood. I think there will be a day in my future I might wanna get another one, especially as they've got different sizes and, two of the three kids in one seat are less likely to wanna kill each other and things like that happen. [00:08:40] Adam Fishman: This is good. Well, I'm glad you did at least get the taste of it for a couple [00:08:43] Adam Fishman: of years though, so. all right. Well, the main topic of discussion is about you did a thing, a very cool thing with AI recently. but not like, "Hey, I built this product with AI," like blah, blah, blah, blah, blah. everybody's doing that. [00:09:00] Adam Fishman: everyone's excited about the health tracking app or the personal assistant or whatever that they're building. But you did something which I think is actually like the perfect Venn diagram of you, you did this, reinforcement learning project to train a model on writing dad jokes. [00:09:21] Adam Fishman: which I love. dad jokes and the idea that you, that you pursued here. So, basically this whole episode's gonna be AI, my AI Corner segment, but taking over the show. I am really thrilled for it. So tell me, what is AI Matey? How does a growth guy end up doing reinforcement learning on a trillion parameter model on your laptop? [00:09:42] Adam Grenier: so I was laid off at Eventbrite about two years ago, which was a really, you know, w- was hard. It was the first time I'd, someone else chose for me to lose, leave a company. [00:09:52] Adam Grenier: and you know, it was also this, like, freakishly great time to be let go and, and have a little bit of severance to buy some time and be able to just go 1000% into everything that was starting to happen on the AI front without having to do a full-time job at the exact same time. I like to say that my, my favorite time where I love tech companies is, like, what I call the zero to two stage. [00:10:16] Adam Grenier: So, like, I love taking things from idea to fruition to product market fit. I, I've done lots of work past that, but what I really love is, like, figuring out that next product market fit opportunity. And so I've always had an interest in, like, product studios. And so that was kind of where my brain was early on. [00:10:34] Adam Grenier: It was just like, I'm gonna build as much as I can, and I want to see if I can build, an AI-first product studio. So not a studio that builds AI products, but rather a studio that is run by AI, right? So, one of the key things that I spend a lot of time in is, like, a typical product studio, one of their superpowers is the ability to prototype and test, and pick what's gonna be the most likely thing to, to push, before they actually put it out and get product market fit and, and get revenue. [00:11:04] Adam Grenier: And, and my hypothesis was, I bet it now w- we can have that same cost actually get us to revenue. So rather than just prototyping the idea, let's actually prototype it all the way through to revenue. And if we can do that at the same volume and pace, like, y- your learnings are actually bound to revenue and real users and things like that. [00:11:23] Adam Grenier: And then the other was kind of, If you do that, you also mean everything you miss on may actually have a value still because it's generating revenue or it's something you could sell, right? You could go onto a secondary market and just be like, "Great, I'm gonna sell this thing that it was cool, but it only makes 10 grand a year. [00:11:39] Adam Grenier: I'll sell it to somebody, make a little money." Which a typical studio, you lose money on everything that's not a hit, right? And so I was just like, "Great, I think this is super interesting." I essentially use that as my excuse to just try every new AI thing that came out. And what I quickly started getting into was, shifting to where the things that I was having the most fun building were the things where I got to learn something, like, freakishly new. [00:12:02] Adam Grenier: something that I'd never done before. As I got deeper into coding, I realized, like, the power of the, API ecosystem and just, like, how much more value the products that we build, uh, already gain because of all of this publicly available stuff that you just tap into from an API perspective. [00:12:22] Adam Grenier: But also a lot of them that I, I think most product managers don't understand, 'cause they hadn't been building the code, there's a lot more we can do with our products by plugging a lot of things in that just already exist. and so I, like, got into the, I wanna learn how to make an API, right? [00:12:36] Adam Grenier: And so I built a goat scream API. I'm happy to talk about that a little bit. [00:12:42] Adam Fishman: I think we're gonna, we're gonna cover that one a little later in the [00:12:44] Adam Grenier: you'll notice a trend when I like to build something new, I don't start with like, "Where's the billion dollars?" I start with like, "What's gonna make me laugh for the next three months, as I go down into this dark cave?" [00:12:58] Adam Grenier: you know, it was like, I'm like, oh, as this person that had some technical skills, now I can do most technical things, with the right model and enough time. I'm gonna go down all these paths to like really understand the we- in, into the weeds. So like the AI side, like I got really into like understanding how to build, you know, a rag and like, why it was done that way and like things like that. [00:13:20] Adam Grenier: And what I found is in a lot of these things, I ended up having to just spend a lot of time in, I think what most people think of as like evals and prompt engineering. but it got really like much more deep than that pretty quick. So for instance, goat screams, right? I basically just wanted to have an API endpoint that you could hit and you get a goat scream back. [00:13:42] Adam Grenier: There, I was inspired by a Owen Wilson whoa, API. Just different versions of him saying whoa. and in doing this, like, you know, I, I went out, I found a bunch of goat scream videos. I scraped the, the clips of the goat screams. I went through a whole bunch of processes to get the right clips, and it just wasn't that huge of a library. [00:14:01] Adam Grenier: A lot of people use the same goat screams over and over again. [00:14:04] Adam Grenier: So I was like, "Well, I'm gonna see if I could generate some goat screams." so I used a bunch of different audio tools and prompted. Like, like some of the best prompts I had were with like ElevenLabs. I found that if you're li- if you say like, "Goat scream," you get like one noise. [00:14:18] Adam Grenier: but if you say like, "A goat screaming like he just stubbed his toe," like you get a way better response. Or like, "A goat screaming like you just snuck up behind him." And so I would just do a bunch of that. the best output ended up being Sora is I would say like a montage of goat screams and it would give the video, but the audio behind it was just a- it was great. [00:14:40] Adam Grenier: It was by far the most just like, oh, it's authentic. It's obviously trained off of other viral, you know, YouTube videos of goat screams. and so I just did all this manual like I'm gonna find the best way to make 1,000 really good quality goat screams. I've never, I don't know if I've said this out loud before. And just how ridiculous the whole thing is. another app I built was, smileplease.a- app, which is still live. Uh, which essentially, like anyone that has ever had to deal with school photos, they're really expensive. If you have to reorder them, they're really expensive. Half the time they're kind of terrible. [00:15:19] Adam Grenier: And so we had some photos that we took of our oldest during COVID, 'cause they didn't have school photos that year, that we just took, like, in our living room. But they, like, stood out as, like, obviously the non-school photo ones. and then we missed a couple years, we had to reorder them, and they were like $300 to order, like, a couple years of school photos. [00:15:34] Adam Grenier: I'm like, "I guarantee I can make better, more realistic school photos than what we just paid 300 bucks with some of the AI tools." But I care. I still want it to look like my kid. I built an app to make, like, school photos. It's, you know, everyone's seen the, like, headshot ones. And it's essentially that, but for school photos. [00:15:52] Adam Grenier: first I spent a lot of time figuring out the right prompts to get the photos that look like they're school photos, right? So, like, I didn't want just, like, me in the '80s. Like, I actually want, [00:16:03] Adam Grenier: like, this is what my grandma's gonna hang up on the wall kind of photos. And so did a whole bunch of just prompt engineering for the environment. [00:16:10] Adam Grenier: And then for, like, how to get it to look like the kid. Do I put five photos in? Do I put one photo in? Like, what angles do they have to be? What quality does it have to be? and so for both the goat scream thing and the photos I built other apps to help me make those decisions. [00:16:27] Adam Grenier: So I built an app for both of them, which was, inspired by something we actually built at Uber at one point, which was essentially a version of Tinder. but you're just quickly saying if you like a goat scream or not, or if this looks like my kid or not. And so I could put like images into a feed, of, one of my kids AI-generated, and I would just quickly like swipe left and right, and I get through all 100 of them in like three minutes, like super fast. [00:16:53] Adam Grenier: Same thing with goat screams. I could just listen to a goat scream, swipe left, swipe right, etc. And so, then I would use that data to then help me prompt tune or model test. So I, for photos especially, I would try all of the different AI models. And then, I would basically create a grid of AI models prompts and be like, "Okay, great, that model is better at smiles. [00:17:16] Adam Grenier: This one's better at eyes. This one's better at like composure, and this one brings in the background better." Like all this kind of stuff. yeah, I just found myself, it's like I want to build something. I want to do it the hard way so I learn, Either it gives me immediate value because I like the school photo thing, or I did something with like school emails that I think both of you and I might have personally built at one point. [00:17:37] Adam Grenier: or it makes me laugh like goat screams. And the more I did this, the more I'm like, "Okay, well, what tools... Instead of me building these things to like grade really quickly, what else is out there?" And that's what kind of slowly started getting me into like the formality of model training. [00:17:54] Adam Fishman: that's a wonderful journey to bring you to model training. and so of course, like, a new model is born like every minute. and the team at, was it Thinking Machines? released their open source, you know, open weight, model. That is the trillion parameter model that I'm referring to, I [00:18:11] Adam Fishman: believe, um, Inkling. [00:18:13] Adam Fishman: And so that's actually relatively new. and you thought like, okay, you know, the next horizon in model training, it's got to be dad jokes. now I [00:18:23] Adam Fishman: understand why you would come up with that idea because like you could be an effective human evaluator of dad jokes as somebody who, one, is like steeped in improv, loves to tell jokes, loves to tell dad jokes, has like an infinite supply. [00:18:42] Adam Fishman: Yeah. you are naturally the best person for the job. you embarked on this journey and the whole thing, just to level set with people, the whole thing cost you like a couple hundred bucks, [00:18:53] Adam Fishman: the whole project. Which is when I think about like everyone talks about, oh, what does it take to train or fine-tune a model? [00:18:59] Adam Fishman: People are like, "Oh my God, it's millions of dollars." You got Anthropic and [00:19:01] Adam Fishman: OpenAI, they're raising billions of dollars to do this model stuff. Certainly what they're doing, creating models from scratch, you know, does cost that kind of money. But there's a lot of tools that help you fine-tune and like post-train models now. [00:19:17] Adam Fishman: So, I guess I'm curious, like just tell me about the journey of getting started doing this and, you picked dad jokes. Were there other things that you were like front runner, uh, options here that you were thinking about? [00:19:29] Adam Grenier: in my like ever long journey of just like learning new things, I like to kind of build exercises for myself. And so, as I kind of crossed the threshold into being like, "Oh, this is more than just prompt engineering." And like when I hear people talk about evals, they're really, they're doing like a pretty light version of what I'm doing for this like random goat project. [00:19:48] Adam Grenier: and so as the deeper I got, it started getting me into like, Stanford reinforcement learning classes or whatever, that I'm like, "Oh my God, I have to listen to this thing seven times in a row and have, an AI agent coach me, like teach me this Greek letter that I don't understand why it's being used." [00:20:05] Adam Grenier: And so basically I'm just like, "Well, I need to have some kind of regular exercise to just like keep learning a little bit more, trying something new every week." I'm sure I saw an, uh, X post or something about this, but like, you know, there's two kind of interesting triggers that are just organic within this space. [00:20:22] Adam Grenier: So number one is that like right now people are launching new models every week, and then number two is th-there's a constant flow of research papers being, released both by these big labs, but also just like there's literally hundreds are coming out every week around some kind of AI or stuff like that. [00:20:38] Adam Grenier: And a lot of people are just like, "Well, just go do the thing they just said they can do." like if a new model comes out or a new paper comes out that says, "Oh, we use this method to train this," the best way to learn is not to go do what Stanford taught three years ago. It's actually to go do the thing that someone just tried to do, and then you can post about it. [00:20:55] Adam Grenier: You're gonna be like, "Okay, yeah, it worked for me," or, "It didn't work for me." and now you actually aren't just learning the thing. You're, you're now part of the conversation. and so I've got like, I don't know, a dozen-ish of these kind of projects of things that I'm training or doing. I've just kind of been wanting one that I would, I wanted to post about that, don't know. I've gotten to a point in my life I'm not really aiming for like virality or a million followers, but more just like, I like to post things that I think are interesting, might start a good conversation. And so, the Inkling one came out, and I, I really like Thinking Labs' vision. you know, there's a variety, there's tons of these businesses building up around models and training and inference and compute and all of this kind of stuff. [00:21:34] Adam Grenier: And there's a few that I think are really honed in on, we're doing this, because there's still a gap in terms of like connecting it to human value, to like what's actually going on with people. And we really want to just make sure that happens. And Thinking Labs is one that's kind of very much built around that. [00:21:50] Adam Grenier: And so I've been following them. So when Inkling came out, I'm like, "Oh, you're great. This is a good chance for me. This model seems pretty accessible. it's half off right now, like as they launched it. let me see, if I can train something against it." And so I literally, I, I, I don't even remember what inspired dad jokes. [00:22:07] Adam Grenier: I think just reading one of their papers, like it talked about, you know, for an open model to be able to see and trace and understand like different effort levels, Started to get into more qualitative stuff, right? Like the ability to actually recognize when allowing a model to think might be better or worse for something that isn't deterministic, right? [00:22:30] Adam Grenier: That isn't a math equation, did the code work, yes or no. But a lot of the things that you and I have built product around our entire lives aren't that. They're humans. They're psychology. They're things like that. I had my own little personal brainstorm for like probably two minutes of just like, well, what's something I can like get a bunch of data around and see if I can train it to do something like qualitative. [00:22:51] Adam Grenier: I feel like there must have been another spark in there somewhere, like I heard a dad joke or something like that's it. the, [00:22:57] Adam Grenier: uh, [00:22:57] Adam Fishman: There was like a [00:22:58] Adam Fishman: collective groan and yeah. Yeah [00:23:02] Adam Grenier: yeah, so, that's kind of what like sparked me to be like, okay, I wanna do one, and this feels like something I could get like, [00:23:07] Adam Grenier: I, I feel like I've learned enough now that I can write about my learning without being a complete moron. Like, I've had actual, like, researchers respond to this, and none of them have said bad things yet. none of them are like, "Wow, you really broke [00:23:20] Adam Grenier: this open, Adam." but no one has, like, thrown tomatoes at me or anything yet for this. [00:23:27] Adam Grenier: and so yeah, so it all just kind of came together and, I think the other element of this is I tend to overcomplicate things, and I want to do projects that are like, full-featured. And so I really force myself to try to think of something that I'm like, "I really can't take this that far." [00:23:41] Adam Fishman: There's a limit. [00:23:43] Adam Grenier: And so dad jokes is like, great, this is something I could actually do fast. [00:23:46] Adam Grenier: I could use artificial data. I could use synthetic data. Nobody's gonna really care. from a training perspective, right, like, I'm sure you're tracking a lot of the, like, buying human data. you want an expert in the loop at some point to be able to grade that stuff. So I'm like, great, [00:24:01] Adam Grenier: this is something I can e- I can pretend to be an expert at, and no one's gonna like, be like, "No, you're not," or, uh, cause it's all subjective, right? [00:24:08] Adam Grenier: And so it just kind of all came together. I'm like, all right, let's see what, what we can [00:24:11] Adam Fishman: So for somebody who's like never even thought about training, obviously I'll link them to the paper that you wrote about this or the, the article you wrote about this, but like, did you need special hardware, special tools? Like what do you even do if you're like, "I don't know where to get started," you want to do something like this? [00:24:29] Adam Grenier: There's a lot in this side of the world that is overly confusing and complex in a way that it, like, it really doesn't have to be, and I think that's what I can do, as well as I think what'll happen a lot over the next few years is just, like, a demystifying a lot of this stuff. [00:24:44] Adam Grenier: but there's a handful of tools. So Thinking Labs has a product called Tinker. That's what I used for this, that allows you to do training. another company I'm, fond of is Prime Intellect. they are doing a lot of open tools around building, like, training environments, and they also have compute accessible, so you can kind of do it all within their ecosystem. [00:25:03] Adam Grenier: there's a company called Unsloth that has a lot. and all three of those have good documentation and cookbooks or good examples, and good direction. I would say Unsloth probably has the best. Like, you can go, they list a lot of the open models, and they say, "Here's how to train it," and, like, they'll walk you through it very step-by-step. [00:25:19] Adam Grenier: all of the frontier models can guide you through it for the most part. I spend a lot of going back and forth asking it why. and so you can do a lot of this stuff and not learn a thing if you try. like it's not that hard to do the training cycle. but to know what happened and then be able to, like, question it and figure out where the cracks are and, like, go back and things like that, I think the more you ask why and the more you're playing in an arena you know, the better off you'll be. [00:25:46] Adam Grenier: But all those are pretty easy. You can do a lot locally, like on your own computer. but for the most part, especially with models this size, like most of us don't have the hardware necessarily to do that. So you do it on the cloud, and, and a lot of them will provide that endpoint. It's, a lot like open router kind of endpoint, which is basically like, "Oh, that's the model I want. [00:26:04] Adam Grenier: I want this one for training. I want this one for judging." You connect the dots and you can run it. it's time-consuming, so like, to get decent training, you want to do, like, thousands of rounds. And so those, rounds, even if they're 30 seconds long, like- [00:26:17] Adam Grenier: For it to fully process and things like that. [00:26:19] Adam Grenier: But, but again, most training is way longer than 30 seconds, right? It's not one half-sentence dad joke. It's usually like, "Hey, go, you know, put this pitch together, pitch it to our CEO, and then, you know, recalibrate it, and then get finance on board." And like that might be like an hour long task. Now do that 1,000 times. [00:26:36] Adam Grenier: Like, that's where the cost like piles up very, very fast, right? And even in my, my writing, I want to actually spend way more time with this model and do like another 100 rounds of training and try a bunch of different things, right? Like one idea I had is like, well, what if I did one round of dad joke training, and then I go do a round of like training it how to do haikus, and then I come back to dad joke? [00:26:58] Adam Grenier: I wonder if it gets better, right? Because a haiku is all about like word economy and thoughtfulness and like that kind of stuff. I'm like, oh, I'm curious. And that's like a real kind of training mechanism to like train other skills to do that kind of stuff. And so $200 was for me to get a very light test to actually see the needle move. [00:27:17] Adam Grenier: for a lot of needs, like that might be enough, right? A lot of like, way to get started and basic things there are much smaller models that you can train. I'm working on like a radio station, which is just like, it's the equivalent of like a friend sitting next to you looking at Twitter while you work, and they just tell you what's going on on Twitter or X. [00:27:37] Adam Grenier: it's more than just X, it's a bunch of other things. But the idea is that like, oh, I just want an audio feed, like sports broadcast that I can like tune into and not have to open X today, but still have the pulse of what's going on there. And so like to do that 24 hours a day gets very expensive. [00:27:52] Adam Grenier: the script writing and the research parts, I like whittled down to being like, "Oh, if I'm on DeepSeek, I can make this pretty efficient." But then it's like, oh, actually, I can take a pretty small version of like a Gemma model, which is Gemini's open model, and I can train it on like the script style I like, and I can actually run that after it's trained on, I have a, a relatively powerful desktop. [00:28:15] Adam Grenier: It's like a gaming PC from a couple years ago, but it's, strong enough that it can run a few models and it's like, oh, I can run that locally and really just be paying electric bill rather than anything. versus even DeepSeek's cheapest model 24 hours a day doing research and writing scripts. [00:28:30] Adam Grenier: It's still like, I don't know, 100 bucks a day and it's like, okay, do I really wanna, before this is even live, be paying, you know, $3,000 a month just to see if it works, when I could do that? And, and so I think there's a lot of those kind of use cases where you're like, oh, if I can get it down to a really cheap model, or it gets a little worse, the cheap model, I bet I could train an open model to do that very specific task that I need it to do, versus OpenAI and Anthropic are trying to train their models on every possible problem of the world to do them all exceptionally, and they're training the whole model. [00:29:05] Adam Grenier: They're not doing post-training, uh, which I'm happy to like talk more about like the whole ecosystem, which is what I'm doing. It's more the like, hey, a model exists. let's tweak its innards a little bit and make it good at this [00:29:17] Adam Fishman: Yeah. I'm gonna jump to the end here, and then I'm gonna ask you if you can, we can attempt the first ever screen share on a Startup Dad podcast. So I want you to show us some things that you did. But I the punchline, no pun intended, is that you trained this model on dad jokes but sort of the skill that it picked up wasn't really humor in the end. [00:29:37] Adam Fishman: what was it instead? what happened? [00:29:39] Adam Grenier: Yeah. So what it got really good at was, getting to the point. and you'll see in, in our demo here, I'm, I'm almost certain this will happen, or if you just go ask any model out there to tell you a dad joke, it'll do one of two things. most of the better ones will tell you ones that exist, right? [00:29:56] Adam Grenier: So if you're like, "Hey, tell me a dad joke about pasta," know, it'll be like, it's an impasta. or, uh, tell me a, a cat joke. It'll be like, "Oh, you know, he was a cheetah," right? Like, it'll tell you ones that everyone knows. if you ask it to make it up or you give it a topic it's, it's never thought about telling it, like, "Tell me a dad joke about the Startup Dad podcast," it'll give you, like, a paragraph. [00:30:18] Adam Grenier: And, like, I don't remember if I wrote this anywhere, but I describe it as, like, telling a joke like my mom would tell it, so a mom joke. Which is, like, you know, I came up in a family of people that are all pretty humorous, and, like, my dad was really funny, and so my brother, sister, and I are all pretty humorous. [00:30:33] Adam Grenier: And then my mom, like, was hilarious because she wasn't, like, naturally, organically funny. She'd just, like, ramble on to try to tell a joke. She'd always explain the joke, why it exists. She'd add some extra part, like a third punchline that was never needed. I think they're all kind of good at mom jokes. [00:30:50] Adam Grenier: And so what I, taught it to do is be like, once you find a punchline, just go with it. Just stick with it. Deliver. Land it. and it got good at that. So, like, the shape feels and looks a lot more like a dad joke, which means when I'm judging it for funny or not funny, it tends to actually be funnier. [00:31:08] Adam Grenier: But when I actually just read it and I'm, like, laying it out next to other, I'm like, it's not actually good at coming up with a really good original joke. but because it lands the shape well, its odds of being funny to me... and I will clarify before we get into this. When I say funny, I mean, like, dad jokes are purposefully not [00:31:30] Adam Grenier: supposed to be funny, right? [00:31:32] Adam Grenier: They're supposed to be bad. And so, like, so it's like, oh, your bad just happens to land in the right amount of cringe that it looks and feels like a dad joke better than most of the other [00:31:44] Adam Fishman: Yep. [00:31:44] Adam Grenier: like, off the shelf [00:31:45] Adam Fishman: Yep. Okay, so let's, let's get into your demo. But before that, you kind of just got into it a little bit, but like, what is the canonical definition of a dad joke for, for you as the expert evaluator? [00:32:00] Adam Fishman: Maybe this is gonna be in the demo. I don't know. [00:32:02] Adam Grenier: Can you see [00:32:03] Adam Fishman: Oh my God, things are happening. Something has been... Oh, yes, yes, yes. Okay. So what are we looking at here? [00:32:09] Adam Grenier: So first I did some deep research. So I, I went to Gemini, I like its deep research capabilities. I don't remember the exact prompt, but it was essentially, "Hey, do deep research on what makes a good dad, dad joke, and come up with a rubric. Come up with like a 50-point rubric or 100-point rubric, of what constitute what tends to be involved. [00:32:28] Adam Grenier: Like go do actual research from, you know, academic research, you know, comedy research, whatever." And so this was the, this was the rubric it essentially came up with, which is the linguistic mechanic, so reliance on obvious wordplay and puns. Like that's the biggest one. Literal interpretation of common phrases, questions and answers, or one-liner format, tame puns. [00:32:49] Adam Grenier: They only violate a language norm, predictable and formulated punchlines, psychological triggers, weaponized anti-humor to teasingly annoy, cog- cog- this is... I, I thought it did [00:33:00] Adam Grenier: great. Cognitive reframing of to regulate emotions, deliver punchlines so unfunny it becomes funny, easing stress and anxiety in awkward situations, fostering parent-child bonding. [00:33:11] Adam Grenier: delivery gets into this. So this one is obviously one that, the inkling is a multimodal model. So one of my like future suggestions of where I would take this is to like maybe try to do a visual element to this to see if it can like react to either like a cartoon or like add someone's facial reaction picture to a joke. [00:33:29] Adam Grenier: and then content bo- so I think the like puns, this like not actually funny, but like it's kinda teasing, cheeky-ish. And then the, the last is that it's actually like, it's not offensive, is the core of the content boundaries, right? It's that like dad jokes that become offensive are actually, they, they kinda aren't dad jokes anymore. [00:33:47] Adam Grenier: like the best of dad jokes at most are gonna be like, know, what do you call a, a brown stick, right? A stick or like some kind of poop joke or something like that. [00:33:56] Adam Fishman: Right [00:33:56] Adam Grenier: Um, all right, [00:33:58] Adam Fishman: Okay, so that was your criteria, your [00:34:00] Adam Grenier: criteria. Yep. [00:34:01] Adam Fishman: criteria. [00:34:02] Adam Grenier: Yep. So the big thing I learned there is that like, one is this like, it's just way too much, right? [00:34:06] Adam Grenier: And, the type of training I was doing, it kinda normalizes all of that into one score. And so it way over-indexed on punts and it way over-indexed on, not being offensive. And so like, neither of those were necessarily bad, but like, being a great pun teller is like a slightly different skill. [00:34:25] Adam Fishman: Mm-hmm. Mm-hmm. Mm-hmm [00:34:27] Adam Grenier: go, go too deep. [00:34:28] Adam Grenier: So here are just actual runs that happened and that were scored by my like judge or by me. Judge being a model that I like gave, "Hey, here are the criteria that we kind of focused on at the end." [00:34:41] Adam Grenier: And so you can come in here and judge these and submit those. I'm asking people to do that because it actually gives me more, feedback that I can do more training on. and you can train your own. let's give it a whirl. Do you have, something you want a, a dad joke generated for? [00:34:53] Adam Fishman: Oh my God. Um, let's see something that's happening in my real life. So my daughter, my teenage daughter is currently at sleepaway camp. Is there any joke that we could up with for sleepaway camp and a teenager? [00:35:10] Adam Grenier: All right. Tell me a dad joke about teenage... And this is the entire prompt, so I don't have any other, like, magic going on in the background. I have 12 models that I, I tested, so this will give us a random one, plus ours will be included. [00:35:21] Adam Grenier: All right, here's what we got. We got, why did the teenage kid pack extra attitude for sleepaway camp? Because they knew it would be a real growth spurt away from home. Why did the teenager pick a flashlight for sleepaway camp? Because their phone needed a break from being their nightlight. [00:35:37] Adam Grenier: Like, that's terrible. W- like, in, in not a good way. Why, why did the teenager pack so much attitude for sleepaway camp? Wait a minute. Because they heard the whole experience was going to be intense. [00:35:49] Adam Fishman: Yeah, [00:35:50] Adam Grenier: I'll, I'm okay with that one. The when teenagers head to sleepaway camp, their parents finally get to enjoy some s'more sleep. [00:36:00] Adam Grenier: Teenagers at sleepaway camp are great at pitching a [00:36:03] Adam Grenier: fit. Why did a teenager refuse to unpack at sleepaway camp? Because he was holding out for a campromise that he wouldn't have to make his bed. [00:36:15] Adam Fishman: That one is real [00:36:16] Adam Fishman: bad [00:36:16] Adam Grenier: right. so m- in my opinion, these first two are [00:36:19] Adam Grenier: terrible. [00:36:20] Adam Grenier: These next four, these next four all, like, start to smell like dad jokes. [00:36:25] Adam Grenier: and again, most people will read these and be like, "These are all [00:36:27] Adam Grenier: bad." but like, [00:36:28] Adam Grenier: if you, if you hit intense right, if you hit pitching a fit, or yeah, whatever, or, like, who that audience is and [00:36:34] Adam Fishman: Yeah. Ooh, Those first two I would put at the very end. they're terrible. I would probably put the third one as maybe number one, I [00:36:43] Adam Fishman: think. I think maybe one, two, three, four is how I might order this. Uh, maybe... [00:36:49] Adam Grenier: One, [00:36:49] Adam Fishman: that's fine. [00:36:50] Adam Fishman: Let's say that. [00:36:50] Adam Grenier: one, two, three, four. All right, we'll do that. [00:36:53] Adam Grenier: All right, my guess is this is mine. This [00:36:56] Adam Fishman: Great at pitching a fit. [00:36:58] Adam Grenier: the, puntune. [00:36:59] Adam Grenier: [00:36:59] Adam Fishman: I was gonna maybe put that as number two. I was wavering, so we're getting [00:37:04] Adam Grenier: Yeah. If th- this... If it had taken like the, the compromise reading it is terrible, but once you say it out loud, it actually landed for me. Like, I feel like the intents and the compromise were the best use of words. so I'm guessing that's mine. I bet... this might actually show us the interesting thing. [00:37:20] Adam Grenier: So number one is Gemini Pro, number two is Sol, number three is Pontoon, four is GLM 5.2, six is Grok, or five is Terra, and 4.5 is Grok. [00:37:34] Adam Fishman: Okay. [00:37:34] Adam Grenier: some interesting [00:37:36] Adam Fishman: Grokus turns out is terrible at dad jokes, even though I thought maybe it would be good. You never know. [00:37:40] Adam Grenier: It, on occasion it gets there, but Gemini almost always wins. across almost all my runs, one of the two Gemini, because I did this one and then, uh, 3.5, [00:37:50] Adam Grenier: Flash. They were always towards the top. it found the balance a little bit better than other ones. the OpenAI ones tended to be towards the bottom, so that... So it's interesting that, you know, Saul popped up well here, but that's the one that often overthinks. It, that was its biggest problem, it would overthink. [00:38:07] Adam Grenier: GLM being in the middle, Grok being in the middle tended to be the case. it dropped to the, the bottom this time. But, you know, Gemini, I don't know about you, but I don't hear it talked about very often. Like, I think everyone respects it, but it's not considered, like, cream of the crop. [00:38:21] Adam Grenier: So [00:38:21] Adam Fishman: Yep. [00:38:21] Adam Grenier: pretty regularly, [00:38:23] Adam Fishman: It's [00:38:23] Adam Grenier: a, pretty big deal ' [00:38:24] Adam Fishman: cause didn't you use Gemini to generate the evaluation criteria? [00:38:28] Adam Grenier: yes, but that's not the, that's not the evaluation criteria used by [00:38:32] Adam Grenier: the end, right? Um, yep. so the training, I trained Inkling, but I used a judge model, right? So like basically like each run that goes through, you give it a score and you feed that score back so it can learn. That's the most generic way I could explain reinforcement learning. that's the thing that like, I'm like, "Oh my God, if like someone I want to hire me that's a PhD someday listens to this, I'm out. [00:38:55] Adam Fishman: No, they'll understand that you're explaining this to a largely dad audience that doesn't [00:39:00] Adam Grenier: That doesn't... [00:39:02] Adam Grenier: And then, uh, I used three judges to make sure it was balanced. and I actually did tests, like I did like, evals against different judges, like how they did and things like that. But like at one point I was using an anthropic model, and one point I was using a Google model as one of the judges. [00:39:18] Adam Grenier: but for my last couple cycles, I took them out for that exact reason, because I wanted to make sure the judges weren't skewing the results. Because they tended to, right? So like if I put Inkling in here as a judge, like, this would probably get a higher rated because it's optimized towards kind of the same thing. [00:39:35] Adam Grenier: The other thing I found really interesting is that like me looking at these, like I c- I, guess at least Pontoon and Gemini, is that, I now read jokes that these different models create, and they all like sound like different dads in my head. Like, so it's, it's like, like in a way, each model's personality is not too dissimilar to like not one dad will tell the same type of dad joke as another one. [00:40:02] Adam Grenier: it kind of became this like weird parallel for me after a while. I was like, "Oh, that's a Gemi- Gemini. You got me again." Like, Saul really stepping it up there. [00:40:13] Adam Grenier: Really, [00:40:13] Adam Grenier: you really over, over-excelled this [00:40:15] Adam Grenier: time. This is the most ridiculous podcast I'm hoping that you've done it out of [00:40:20] Adam Fishman: I think so. I think [00:40:21] Adam Fishman: 150 I think, I think it is. okay, so the way that this works, just to, just to rewind is, so this goes out, you give it a, a prompt, very simple, straightforward prompt. It goes and asks a bunch of models. Now, does it randomize the models that it asks? So it asks six models, but will it ask a different [00:40:38] Adam Grenier: yeah. In this, in this arena, it's randomizing out of the 12 that I have, [00:40:43] Adam Grenier: so these. So two Gemini models, two OpenAI models, Terra, two Anthropic models, and then Kimi, GLM, and Grok. [00:40:57] Adam Fishman: Yeah. Interesting. So it's, it's funny because these are the scores, right? And it's funny because I think of Claude and Anthropic models as being pretty good writers. Like I think of Fable, Fable's generally towards the top, you know, third or whatever, but like OpenAI stuff, not. and I generally think it's good at writing, but it's, it's good at certain types of writing. [00:41:24] Adam Fishman: Clearly not dad jokes, which have a very specific requirement to it. So [00:41:28] Adam Grenier: and so what's interesting is that pre, Goat Screams, one of the other things I wanted to build just to build it was a, uh, Visual Studio extension. 'Cause as I started getting IDEs, I'm like, "Oh my gosh, there's a whole marketplace of extensions." [00:41:41] Adam Grenier: I'm a marketplace guy, you know that. For the audience, all I do, I, I like marketplaces, and so my brain is like, "Oh my God, There's this whole new wave of humans that are gonna become users of," so I do think revenue at times. But a whole big wave of people are gonna be IDE users. That means the marketplace of extensions in IDEs may be like its entirely new environment, for, like, non just engineering first things. [00:42:05] Adam Grenier: So I was like, "Great, I wanna build one. I wanna use an, a model, do that." So I, built, one that does your commit. like it generates a commit, phrase for you, and it's in a haiku. So it's a haiku commit. It basically analyzes whatever you're about to commit, and it rewrites it as a haiku. that was one of the hardest parts of it, was getting it to write a good haiku because, models don't do good with, like, word count or character count, because they don't look at it that right. They look at what is most likely to be next. so a space might be one of those. [00:42:32] Adam Grenier: They don't actually think about it as, like, different words. So getting the, like, five seven five structure down right was, like, really hard. and so I think that was, like, one of my first instincts of like, oh, writing with, like, a very specific boundary, it starts to break some of what is good and what's not good. [00:42:48] Adam Grenier: If you give it, like, something generic, boundaries, I can go all day long. I have the same thing with this radio thing I'm building, is that, like, if I'm like, "Oh, make a one-minute clip," like, they're all over the place. They're not, like, 10 minutes. They're not 10 seconds. But the ability to land within, like, 10 seconds, whether I give it characters, time, or word count, like, it's rough unless I ask it to, like, do, like, a Python script to check its [00:43:10] Adam Fishman: Yeah. So the whole, game of training a model is really, like, based on the reward system. So for people who, don't know, like the reward, you kind of covered this in a very, like, high-level way, but it's basically like it's a score that tells the model what it did well, and you consistently feed that back into itself so that it, learns. [00:43:35] Adam Fishman: you went through a bunch of different reward designs here, and they all fail in like a slightly different nuanced way. So, I would love to hear, like, how you did the reward systems and, like, what your favorite failure modes were. [00:43:50] Adam Grenier: think this goes a little bit back to me being able to learn from this specific topic of dad jokes. I think I was able to learn reward design way better than if I did something I either didn't have a strong opinion about, or if I did it on something that was really deterministic, right? [00:44:14] Adam Grenier: because if it was broken, I'd have to, like, figure out why, or there might be, like, a really obvious answer. Where most of the time I would do a run and I'd be like It just doesn't feel right. [00:44:25] Adam Fishman: Mm-hmm. [00:44:26] Adam Grenier: And then I would do this. I would just talk for a while. I'd be like, "Yeah, that, this isn't weird." Like, I don't know, it just isn't like landing fast enough for me. [00:44:35] Adam Grenier: Or when I read these ones and I read these ones, and I do that enough, I'd be like, oh, it seems like, I think it's skipping this element of what makes a dad joke. And so then what I would do is I would take, you know, my original rubric and I'd be like, okay, great. Well, instead of everything being two points, so the first cycle was like, if I actually look at how each part of the rubric was scored, they were all getting perfect scores on everything to do with safety, right? [00:45:01] Adam Grenier: With, is it offensive or not? whether that's because they've been trained on like, you know, red team, like not to cuss, not to say bad things, whatever, plus dad jokes mean, you know, innocent or whatever. But because that was a quarter of my entire rubric they were all acing, it actually made the ability to see movement in those other elements harder. [00:45:22] Adam Grenier: Because they were already at like a really high waterline. So basically, I was like, well, I could just take that out. Like, I could not give points for that, or maybe I just like, maybe I give five total points for the safety tone, that type of stuff. That way now, like the pun side of it, and same thing with like the visual response. [00:45:39] Adam Grenier: I'm like, okay, well it doesn't actually know how to do that, and I don't know how to score that well. I can remove that. So now I'm actually gonna give more weight to is it doing wordplay, and is it doing in a way that's gonna kind of catch you by surprise, in like a cheesy way? And so it's like, okay, great. [00:45:55] Adam Grenier: So that was kind of like cycle one. And, length was a very clear next one. So it's like, oh, okay, the ones that are funny are all like, they both feel funny and they land for me, they get to it faster. So like I said, when we looked at those comparisons, it's like if that intense joke was put into the puntoon row, it would've easily been number one. [00:46:14] Adam Grenier: Because it's like, oh man, they just did that in five words, and it was like, I didn't expect it. It hit me. It was cheesy. All right, that's it. That's a great dad joke. So I'm like, okay, great. I'm gonna make word economy, one, it's easy to judge because now I can make that a deterministic output instead of a, an AI model judging if it was long or short. [00:46:34] Adam Grenier: I could be like, great, count characters. Is it above this? Is it below this? Give them that score back. And then what started to happen is I'd get these like the joke would like get cut in half, but I was still rewarding puns. And so I would get like, "Oh, I have an avocado that's a friend and it's the pits. [00:46:50] Adam Grenier: It's really like, it's getting me green and it's, it likes my good fat," [00:46:56] Adam Grenier: or something. And so it would be short, but it would have like three different puns in it, which took away from all three of them. I was just like, three jokes doesn't actually make this better. [00:47:08] Adam Grenier: It makes it worse. And so then I'm like, oh, if I'm rewarding length and I'm rewarding the use of puns, it learned, it hacked basic- So this is reward hacking at its finest, is it hacked the fact that like, oh, if I put two puns, I get 100% on the pun category every time. because a AI judge just says, "Does this have a pun?" [00:47:32] Adam Grenier: And sometimes maybe it doesn't know if it's a good pun or not, but when there's two of them, it never misses. ' [00:47:37] Adam Fishman: 'cause it's got, [00:47:38] Adam Grenier: a [00:47:38] Adam Grenier: pun. [00:47:39] Adam Grenier: Yes. 10 out of 10, 5 out [00:47:41] Adam Fishman: It's like, this is like Goodhart's law in action, right? Like this is, for a kid, this is like I put a sticker chart on the fridge and for my kid to get cleaner, and they don't actually do that. They just get better at doing a thing that gets them a sticker. [00:47:57] Adam Grenier: the product version of this, right, is the like, we just got feedback from our customers that they don't love our product, or they're, they don't feel engaged enough. And so you're like, "Oh, well, the people that do love it, they all click like," right? [00:48:13] Adam Grenier: So we're gonna find 12 hacks to make everybody click like 500 times. And now likes actually don't mean anything 'Cause it's like, oh, I took the thing and I actually like bastardized it enough that it's now no longer... I mean, that's Goodhart's law. It's now no longer effective as a measurement [00:48:30] Adam Fishman: So you did all this work and then you started getting Goodharted basically. The Gooder- Goodhart's law is, essentially like basically any metric that you measure ceases being valuable as a metric because people will learn to game the system. And so AI learns to game the system, right? [00:48:45] Adam Fishman: And we've seen like things written up in The New York Times about AI like gaming the system, because it's goal seeking, right? It's goal seeking behavior and it's like, what's the fastest way for me to get to the goal? So when this happened to you, was this kind of disheartening to you or were you like, "Oh, this is fascinating. [00:49:03] Adam Fishman: I learned something here." Like what was your reaction to [00:49:05] Adam Grenier: The, I mean both, right? So like, again, as I mentioned, they took a while. a handful of the runs that I would do, I would just start before I go to bed, and then I'd wake up and it'd be done or broken. But it'd be finished and be like, "Okay, great, I just spent, you know, 75 bucks." Which again, if I were a lab or a business, no big deal. [00:49:23] Adam Grenier: But I'm like, I'm not making any money off of this unless, except for the, podcast royalties I'll get from this episode. Yeah. so it's just like, man, 75 bucks for like one night of runs, and it made the jokes worse. Like these are way worse. And I'd be like kind of pissed. But then once I'd be like do the talking through and figuring out, I'd be like, "Oh, that's awesome." [00:49:45] Adam Grenier: Like me learning reward hacking by being reward hacked seven times or six times in a row, by the fifth and sixth, like [00:49:54] Adam Grenier: I'd be able to do like three cycles instead of [00:49:56] Adam Grenier: 100 cycles and be like, "Ah, [00:49:57] Adam Fishman: and that's [00:49:58] Adam Grenier: crap, there it is." [00:49:59] Adam Fishman: 75 bucks right there, right? That [00:50:01] Adam Grenier: Yes. Yeah. [00:50:02] Adam Grenier: Yeah, 100%, right? [00:50:03] Adam Grenier: And so that's where doing something like this, you both learn how to do it better and well and why it matters. And this is the kind of stuff that, one of the head AI guys at Bastion, which is one of the, the big imprints companies, just was, had a really great podcast the other day about like how to get into RL and all that, that kind of stuff and what he would do different now. [00:50:22] Adam Grenier: And his big thing was just like just go do it. first you learn all of the, hard math, and then you learn all of the data science, and then you learn all of the like, machine learning, and then you can start getting into model training, then you can get into reinforcement learning. [00:50:39] Adam Grenier: Or you can sit there with Claude and Onslaught and do it tonight, and you're gonna learn six things that most PhDs don't learn until they're like sixth year into this. [00:50:48] Adam Fishman: trying to learn how to play basketball by just watching a bunch of videos and reading a book about basketball, you know? [00:50:54] Adam Fishman: Like, yeah. [00:50:55] Adam Grenier: time we play [00:50:57] Adam Fishman: You're gonna be so good at this. [00:51:00] Adam Grenier: My, based on my TikTok feed, I now believe I can beat you in basketball. [00:51:05] Adam Fishman: Um, If I recall, the thing that ended up being like the greatest predictor of the humor in a dad joke was Was it word count? [00:51:13] Adam Grenier: Yeah, so the length one was by far, of all of those things, length was the one that like tamed everybody, so one of the other cycles I went through, it's not just the shortest wins, it's like, thoughtfully short. So we saw like the Gemini one that responded. [00:51:29] Adam Grenier: It was still a full sentence, but it wasn't so long that it lost the ability to hit for both of us, but anytime I didn't use length as a driver or a reward, nothing else moved the needle as much as just like, "Hey, just get to the point." and it's not just length, it's also about like I talked about, like using multiple jokes, like by saying like the length matters within reason, and hitting on one joke matters, staying on one topic matters, you're essentially training like, oh, the Venn diagram of those things is like, just get to the [00:52:04] Adam Grenier: point, [00:52:05] Adam Grenier: so for instance, like one of the things I think I appreciate more now about dad jokes than I did in the past was that like the best dad jokes don't usually have a setup in a traditional joke sense, because that was a cycle, especially in the first couple, a lot of them were a very clear two-sentence structure. [00:52:23] Adam Grenier: why did the X do the Y? Because of the Z, right? And like that structure was used over and over again, and they just never landed as much as, Oh, I have a friend that's an X, Z," and it's like, "Oh man, that thing is like hilarious. I did not expect that. Where do I go," right? see, this is my, what happens when you look at [00:52:41] Adam Grenier: model training too long. You [00:52:43] Adam Fishman: W- so you just mentioned that, like, you learned something about, you know, joke telling in this, but, like, did you find in the course of this project, did this do anything to your sense of humor? are you able to deliver a better dad joke now? Has it changed nothing [00:52:59] Adam Grenier: Oh, no. [00:53:00] Adam Grenier: No, not at all. yeah, my problem is clearly I think a lot about things. my dad joke hits are when I didn't have any plan or expectation to tell a dad joke, I do those great. Like, I'm an improviser. I'm really good on the spot. if anything, if I'm like, "I'm gonna tell a dad joke," I'm out. [00:53:18] Adam Grenier: 'Cause now I'm like, let me go through my 50-point rubric. [00:53:20] Adam Fishman: Right [00:53:21] Adam Grenier: let me good heart myself a few times. and now I've got it." [00:53:24] Adam Fishman: is, this is also my lesson for posting on LinkedIn, by the way. If I just like YOLO something out there, it goes viral. Every time I try to think about a very thoughtful post, it gets like two likes [00:53:35] Adam Fishman: and, and [00:53:35] Adam Fishman: 50 views. [00:53:37] Adam Grenier: so much time [00:53:38] Adam Fishman: I know. I'm just gonna YOLO it from my phone. okay, so I have to ask you on the judging side. [00:53:45] Adam Fishman: when I had you on the show last time, you told me that one of your favorite things to do was to get your kids involved in like pranking your wife. [00:53:51] Adam Fishman: her... [00:53:52] Adam Grenier: not my favorite. It's a [00:53:53] Adam Grenier: strategy ' [00:53:54] Adam Fishman: 'cause sh- she [00:53:55] Adam Grenier: is [00:53:55] Adam Grenier: not fond of me [00:53:56] Adam Fishman: She didn't like it. [00:53:57] Adam Fishman: when you did it, so therefore your kids, it was fine. did you draft any of the kids as judges in the dad joke, training? [00:54:05] Adam Fishman: Like did you bring in your teenager or your middle kid or, or [00:54:08] Adam Grenier: y- y- you know, not as much. I 'cause everything here in this post and model that's trained is purely me and AI models, right? And so one of the reasons I wanted to post it live is like, great, I want to get other people to go submit stuff, make up stuff, judge it, and things like that. [00:54:26] Adam Grenier: So now I'm getting a human element in it. my idea is that, like, I'll compare that against what I've already created against my kids. now I've got like three different kind of versions. I've got the AI and one human, the training on a whole bunch of humans, and then just like, but the real judge should be them. [00:54:44] Adam Grenier: I think that could also be a really interesting way to add the multimodality to it, 'cause I could probably create something easy that takes each of these jokes and, like, reads them out loud, and I just record their reactions to [00:54:56] Adam Grenier: them and see if I could train it to [00:54:59] Adam Fishman: how high do the eyes roll in the head, right? Yeah, exactly. Exactly. okay, so, there's a thing that I didn't ask you about yet that I wanted to, which, you know, you sort of called me out on this as the teaser that I was gonna put in this, which is in the middle of this project, you fired somebody, [00:55:16] Adam Grenier: Угу [00:55:17] Adam Fishman: rather. [00:55:18] Adam Fishman: So tell, tell me about the great firing in the dad joke project. what did you do? [00:55:23] Adam Grenier: Yeah. So, I Was doing a training cycle and two things happened at once. the price for the judges started to spike. Just like the total, like, billing prices started to spike. and then two, it just slowed down, like, a lot. And so in going through my, like, arena of who's the best judge, uh, one that did really well was actually the new, the Spark model from Meta. [00:55:50] Adam Grenier: and it was relatively affordable. so I'm like, great, it judges better than others in its price range. I'm gonna throw it in the mix. but in the judging, it thought way more, and it took way longer. And so by doing that, it was using way more tokens than my other judges. [00:56:05] Adam Grenier: And I- y- you can cap that, but for some of these models, as you'll see in the writing, like, different caps you put on things can influence it in weird ways that you don't necessarily want. And I wanted consistency was important for me. as with, humans that I have to deal with, if they're too slow and they're costing me too much money, [00:56:20] Adam Fishman: They got to go. [00:56:22] Adam Grenier: they're out. [00:56:23] Adam Grenier: The, uh, and so I fired it, but I actually brought it into the arena, so I don't think I mentioned it when I was saying it out loud. And it did middling. but there, so I've got my, all the jokes that were judged broken into a few different categories. I've got like, you know, basic topics like giraffes and avocados and things like that. [00:56:42] Adam Grenier: I've got esoteric things, like Newton in physics or, you know, things like that. It's like, oh, you probably don't wanna hear a joke about this maybe. and then, esoteric topics, it won. So Muse popped to the top for esoteric. [00:56:54] Adam Grenier: So, because I think that problem of overthinking, some of the models that have all of the world's knowledge in them may spend a bit too much time thinking about these, like, really esoteric topics. so yeah, so fired Muse, but then rehired them as a contractor, but uh, to come in and, still respond to my [00:57:11] Adam Fishman: Okay. Okay, good. okay, so, at the end of the day, you've got a dozen models like fighting over each other, evaluating your dad jokes, like, But if you boil this all down, like you created a dad joke benchmark. But I do think that there are probably some real practical takeaways for people that aren't working on dad jokes, but are working on other stuff. [00:57:33] Adam Fishman: if you had to like extrapolate to like a few lessons that you took away, and some of them you've written about in the, in the post, which we'll share. But like what did you learn in doing this that kind of extends beyond the dad joke, umbrella? [00:57:47] Adam Grenier: Yeah, so I think, the being in the weeds of the traces of the like, being in the data, not just in the, like, outputs of the data, which I think, you know, a big difference between product marketing and model training data is that, like, one, you have way more of it. You can get super granular. [00:58:07] Adam Grenier: Two, you can see what they're thinking, not just what they're doing. and so getting, like, into all of that and looking at it, like, qualitatively makes a big difference. And I think it's, easy in our lives to just, like, move everything to quantitative space. but being in the weeds is how I was able to, like, start sniffing these different reward hacks and being able to understand, like, what I'm optimizing for, and even understanding what I wanted out of the end. [00:58:30] Adam Grenier: the other key thing for me is that, okay, with a couple hundred dollars, and that's me being sloppy and not great at this yet, right? Like, I could either make it more efficient or I could spend more if I wanted to make it better. I was able to take something and get it to a zone that I really liked. [00:58:46] Adam Grenier: if you think about the amount of qualitative things in our lives up until this moment in product and marketing, it's almost everything you do. It's human psychology. It's people making decisions under stress in environments that you don't have control over. it's things like in different spaces and all that kind of stuff. [00:59:05] Adam Grenier: And it's like there's so much of that world that is still about the quality and what quality you want to bring as a product or as a brand, and how do you land that, and how do you actually do that? And I think, being able to take the actual people that are building that product, that are spending time with those customers and those users and allowing them to be part of this process, I think will drastically increase. [00:59:31] Adam Grenier: It's way more accessible than it's made out to be right now for us to actually bring this type of training way closer to the user, to the person that's getting the value. Whereas right now it goes through this huge cycle of like, oh, we have to go fake people to, like, get their data to then go train models, to then go be influenced somewhere else, to then be used generically by the company that actually has the relationship with the user to serve it to them. [00:59:56] Adam Grenier: And so it goes in that huge cycle versus just being a one-to-one relationship, being like, "Hey, that's our product." The people that are building this product, spending time with the users should probably be the ones that are helping influence this, and I, I think they can way more than, like, prompt engineering. [01:00:09] Adam Fishman: Now that you've opened this up to, and you publicized it, and I guess some people are probably playing around with it, what are you learning now? Like where has the project gone? What kind of direction has it taken since other people can get in there and start, giving scores and evaluations and stuff? [01:00:26] Adam Grenier: Yeah, so I think one of the big ones we've touched on a little bit, which is, some people don't appreciate a bad joke. [01:00:36] Adam Grenier: in not a bad way, but I've, I've had a handful of folks being like, "Actually, all of these are [01:00:39] Adam Grenier: terrible." I'm like, Yes. [01:00:41] Adam Grenier: but, but are they terrible like, in the right way?" Right? so, so I think that's one thing I've learned. I think one of the things that I, I struggle with with a lot of the, the exploration I do is that like I naturally now wanna go do 10 new things with this. [01:00:57] Adam Grenier: But then there's also 10 other things that aren't this that I wanna do. And so for something like this that's more of a pet project, that decision's a little harder, right? so like we, came up with a couple ideas here and that seem like great, I wanna go train that. but it's like I'm gonna keep training and spending more and more money on this model that again, is probably not something that's gonna be the, the big revenue break or the thing that's gonna land me the job or whatever it might be in the future. [01:01:20] Adam Grenier: my hope is I'll still do maybe, I would like to do something multimodal with this partially because Inkling is a multimodal model and like, it's, a focus of that model. And so I, I wanna do something there, and then I wanna do something with the human reviews that I'm getting right now. [01:01:34] Adam Grenier: so those are the two things that I do wanna like close the book on and have kind of a follow-up of like, okay, great, with those two ingredients, what else do we get out of this that you don't just get out of one guy in his basement telling himself dad [01:01:46] Adam Fishman: Yep. Yep. Um, so I want to ask you about your favorite joke that came out of this, but I don't think that came from a model. [01:01:57] Adam Grenier: I have a range [01:01:57] Adam Fishman: tell me about your, some of your favorites and like what the best one was. [01:02:01] Adam Grenier: so one was the, the impasta joke. So what do you call a fake pasta? An impa- an impasta. The, uh, I've heard that joke a thousand times. So one of the things I ended up doing was, adding jokes as a reference because, as I mentioned, a lot of the models would just tell existing jokes rather than writing new ones. [01:02:21] Adam Grenier: so basically I made, a big pool of existing top jokes on the topics that I chose, and graded those negatively if they used an existing one and, and added the word, like, create an original joke. but like that was one of them that I would like in like 50 that I would just constantly review I basically had this set of 50 jokes that I would review over and over and over and over again, with the different models so that I'd have a consistent, like test suite. [01:02:48] Adam Grenier: And that was one of them, and it made me grin or smile all 50 times, right? I was like, "Okay, that is classic. It hits. It is like the pinnacle." So I, I think my appreciation of that. One night, I was like going down this rabbit hole with, I think it was Claude was, guiding me at the time, around like, okay, [01:03:09] Adam Grenier: if I add the expert side of this, but I did it not in reinforcement learning style, but in more of a, fine-tuning style, but I did both of those in the same session. Like, is there a name for that when you use like two different model optimization methods like in one? [01:03:27] Adam Grenier: And it's like, yes, and this one has a name, and it's super boring. It's this. when it's like it doesn't have a fun name, I was like, "We need to give it a fun name." And then instead of doing that, I just started writing dad jokes about, reinforcement learning. And so, I don't even know if these are gonna make sense to anybody else but someone that's really deep into this, [01:03:48] Adam Grenier: um- [01:03:49] Adam Fishman: the engineers at, at, uh, [01:03:50] Adam Grenier: Let's see. Yeah, I'm, I'm trying to remember. There were two of them. One was... basically it's, you take a, group of answers and you make their whole average, and you tell the model for each one how much above or below average was it. And that, that is an easier way for a model to know if it was good or bad than giving it a number. [01:04:11] Adam Grenier: for various certain things. And so that's, group relative's PO. I can't remember PO right now. this is the difference between someone that studies this forever versus someone that tells dad jokes about it. and then, DPO it's when you say like, "Hey, here are 10 and this is the good one." [01:04:27] Adam Grenier: Like, this is what good looks like, these are not. This one is good, so you give it an example. And so I wrote the joke, two optimizations walk into a bar. GRPO orders a flight, but together they're just average. DPO picks their favorite, still doesn't like it, and won't shut up about it. And so again, I, this is like late at night and I'm like writing this. [01:04:48] Adam Grenier: I'm like, "I did it. I've hit both my own jumping the shark [01:04:53] Adam Fishman: Yeah [01:04:53] Adam Grenier: also my like I think I'm qualified to write this now." [01:04:57] Adam Fishman: I, I think it's a great joke, I mean, personally. [01:05:01] Adam Fishman: let's take that one right to the punchline, in SF. So I wanna wrap up. I learned a lot in this conversation. you know, three years ago when we did this show, maybe not surprising to people, but your parting advice was that a good sense of humor is one of the best skills that you can teach your kid. [01:05:20] Adam Fishman: and then you spent a week discovering that you couldn't exactly teach a sense of humor to a machine. You could only teach it to, like, commit, and be direct and kinda get to the point. So I'm wondering, did that change whether or not you think that humor can be taught? Can you still teach, uh, one of your kids humor? [01:05:43] Adam Grenier: Yeah, I think so. I think it makes me appreciate both in humans and in models, like the suite of ingredients matter, right? So we looked at that rubric, right? It's not just is the wordplay good. It's not just, you know, did you keep it inoffensive? but it's also like who was it delivered to will make a big difference on if it was funny or not. [01:06:06] Adam Grenier: you hit the punchline will matter differently based on who the person is. When you deliver it, it might work in one environment, and then you, a week later in a different environment it won't work. I believe that each one of those angles is a trainable element in a model. [01:06:22] Adam Grenier: There's still gonna always be art, and I think part of what makes the best models in the world great is they're getting better and better at knowing how to use the suite of those things and figure out the moment and, actually be able to kind of like adapt around various pieces. and so I think just like teaching my kids how to have a good sense of humor, it's a collection of years and years and years of me making a joke, them seeing people react to it, good or bad, mostly bad. [01:06:50] Adam Grenier: them telling a joke and other people reacting to it, me explaining jokes. So my, my oldest daughter is really into musicals, and before I introduced it to her she listened to, the entire album of Book of Mormon. And I was like, "What'd you think?" And I don't know if you know Book of Mormon, but it's, pretty offensive, right? [01:07:08] Adam Grenier: It's, a- and so, like, and as it goes, you know, down some pretty deep topics that, your average dad doesn't talk about with their 15-year-old daughter. And she's like, "It was interesting." And so, I was like, I literally like had an entire conversation with her around just like, the style of humor is where you're actually like you use being offensive to make a point that is way bigger than the joke in and of itself. [01:07:34] Adam Grenier: And like that type of stuff's like, I naturally have these moments where I'm actually like taking a beat to be like, "Look, you were just introduced something that's gonna make people uncomfortable." And you could either look at it as offensive or inoffensive or you can zoom out and be like, "Well, what are they talking about here?" [01:07:49] Adam Grenier: a lot of the themes that are covered in Book of Mormon are, are pretty, like, legit conversations that sh- people should be having. And it's like that's what's important. It's not that moment that that thing made you kind of like cringe, but you cringing made you think about it. and those are the kinds of things that it's like could I do that with a model? [01:08:05] Adam Grenier: Yeah, if I had enough time and enough money and enough energy. I believe so. I don't see that as dark. I know a lot of people in the world right now do. I see it more as exciting 'cause I want these things to enable and make everybody more fulfilled. And if it means I can train a model that, you know, can catch me off guard and make me laugh about, intents, [01:08:25] Adam Fishman: A-awesome. Or impo- impasta. okay. To end, two questions for you. one is, what's next for DadBench? Have you started reaching out to Anthropic and OpenAI to see if they'll include it in their tables of, uh, model evaluators? but [01:08:41] Adam Fishman: seriously, [01:08:41] Adam Grenier: No one's opened the email yet? [01:08:44] Adam Fishman: I'm shocked. Maybe someday. Yeah. but it sounds like you said doing like multimodal things is kind of interesting to you, and then obviously like taking the project into other learning directions and stuff like that. It seems like your goal is just continue learning and trying new stuff here. [01:09:01] Adam Grenier: Yeah, I, I want to learn and try and, like, I think the hypothesis that I'm really leaning in on is, you know, data science in, building models to do bid optimization in ads evolved from, you know, spend a little more on a click, spend a little less on a click, you know, that type of stuff, to product optimization and actually, like, putting, like, real, models and machine learning to both make decisions in the product and, help product teams make decisions. [01:09:28] Adam Grenier: Each of those had a curve where they became more and more accessible, and much more the hands of product managers and marketers. And I think this is gonna go down that same path. I think you're already seeing the cost of models being a very real issue in businesses, and the effectiveness, and much more importantly for a lot of the bigger companies, like privacy, right? [01:09:48] Adam Grenier: And, like, where is this data going? Who's training on this data? And the more those things become important, and the more this stuff becomes accessible, I think the more you're gonna be like, "Okay, great. If there is a tool like Statsig that a product manager can be doing what they think is model optimization, and they're actually doing reinforcement learning," I don't see that being unrealistic over the next couple of years. [01:10:09] Adam Grenier: And so that's kind of where I'm circling, is, like, I wanna do stuff to see if, if I can do this, Can I take that whole process? What's the product that could have made this accessible to Adam, to other people, right? So that's kind of like the underlying of, like, where this exploration is going for me. [01:10:24] Adam Grenier: It's like I wanna learn, but I still wanna learn to connect it back to the worlds that I, [01:10:29] Adam Fishman: Yeah. Makes sense. So in that note, how can people who hear about this project or watch or listen to this pod, be helpful to you in your current journey? [01:10:40] Adam Grenier: I just, like, enjoy talking about this stuff with folks that are curious on either end of the, they're a PhD and they're trying to figure out how to actually connect this to more, of the day-to-day product world versus just the model training or the science and coding side of the world. [01:10:57] Adam Grenier: or, like, product folks that are like, "Okay, great," like, "I wanna be thinking about how we start leveraging this," or, "We are starting to leverage this. How do we do more of it?" I just love jamming with people and, like, digging into it, sharing with them what I've learned, challenging them, asking them if they've got something they wanna try and we do it [01:11:12] Adam Grenier: together, all that kind of [01:11:13] Adam Fishman: Okay. Well, with that, one other thing that I wanted to come back to, which I don't think we have time for, unfortunately, but three years ago when we did this show, I promised you that someday I would invite you back to read me the Sesame Street emotions book and to do it with all the voices. I didn't prompt you for this, but I don't know if you have the Sesame Street emotions book with you, but my guess [01:11:37] Adam Grenier: I, A, I don't have it [01:11:38] Adam Grenier: handy. B, I don't know that I have it anymore. but I [01:11:41] Adam Fishman: so s- [01:11:42] Adam Grenier: How about this? I'll record it [01:11:44] Adam Fishman: yes. Okay [01:11:46] Adam Grenier: I'll probably put like an AI animation on [01:11:48] Adam Fishman: Yes. And we will append this as a, a [01:11:51] Adam Fishman: bonus cut to people who make it this far into the, into the episode. yeah, I wanna keep my promises, so I will open the floor to [01:11:59] Adam Fishman: you to do that. So, uh, thank you so much, Adam, for joining me today. this was awesome. I'm glad that you are thinking about how to make this stuff more accessible for non-technical, non-PhD people, dads, and also just having fun with it. [01:12:15] Adam Fishman: Like, it doesn't have to be all serious and cyber hacking all the time. We can occasionally fine-tune a model to get better at telling dad jokes. So I appreciate you doing that, and the goat screams. and we will link to all of these things in the show notes, [01:12:30] Adam Grenier: I, I love it. Someday I'll actually do a more formal write-up and announcement of the goat screams, 'cause it, that one I did annoy my [01:12:36] Adam Grenier: family [01:12:37] Adam Fishman: bet. I'll bet ' [01:12:38] Adam Grenier: Cause I'd just purposely sit around them hitting, like, call goat [01:12:42] Adam Fishman: Yeah. Well, now you could probably hook it up to like a Google Home or a Nest device [01:12:48] Adam Grenier: 100%. [01:12:48] Adam Fishman: something like that and just have it scream every now and then, so. [01:12:51] Adam Grenier: Oh, 100%. [01:12:52] Adam Fishman: right. Well, thank you so much, [01:12:53] Adam Grenier: Awesome. Thank you, Adam. [01:12:54] Adam Fishman: great. Talk to you. [01:12:56] Adam Grenier: Yep, take care. [01:12:57] Adam Fishman: Thank you for listening to today's conversation with Adam Grenier. You can subscribe and watch the show on YouTube or wherever you get your podcasts. [01:13:05] Adam Fishman: Visit www.startupdadpod.com to learn more and browse past episodes. Thanks for listening and see you next week.