Latest / Chaos Agents / Character.AI's bankruptcy, medical AI's rubric problem, and the Parsimony coding benchmark

Transcript

Character.AI's bankruptcy, medical AI's rubric problem, and the Parsimony coding benchmark · Chaos Agents

• 0:00
  All right, we are back. Welcome.
• 0:03
  We are back. Welcome. Hey,
• 0:03
  Hey Sara. ~
• 0:06
  Becca.
• 0:07
  Hey. welcome to Chaos Agents. We are an AI News podcast curated by
• 0:12
  an agent named Eddie and then questioned,
• 0:15
  debated, and occasionally yelled at by human hosts Sara Chipps and Becca Lewy
• 0:20
  I feel like we rarely yell.
• 0:23
  We we don't yell. It's more like it's more if we ever yell,
• 0:28
  it's not literally yelling, it's just implied yelling and it's it's in jest,
• 0:33
  you know. It's
• 0:34
  It's in jazz and it's in love.
• 0:36
  in love. It's 'cause Eddie can take it.
• 0:38
  Eddie likes that kind of banter,
• 0:40
  you know?
• 0:41
  Yeah, and if you want to get the newsletter this podcast is based off of,
• 0:46
  you can go to chaosagents.ai forward slash listen to sign up and you
• 0:50
  can see the links from today.
• 0:52
  Amazing. Sara, let's get into it 'cause I know we are both super bummed
• 0:56
  to talk about.
• 0:58
  This is fascinating. And yeah,
• 0:59
  I have so many things to say. So many things to say.
• 1:00
  Okay, I'm gonna set the scene and
• 1:02
  then let's jump and then you can jump right in.
• 1:05
  Okay.
• 1:06
  okay, so the first headline, I'll just read the headline first.
• 1:10
  It's Character AI filed for bankruptcy.
• 1:12
  What happens to millions of AI companions now?
• 1:15
  So this is a story about character AI filing for bankruptcy.
• 1:21
  Yes. They're a company that does does like fictional AI characters
• 1:26
  for companionship and scene building.
• 1:29
  You can kind of identify what character you want to talk to,
• 1:33
  be friends with. It could be in fiction,
• 1:34
  you can make your own, that kind of thing.
• 1:36
  Which
• 1:36
  leads me to my first fun question of the day.
• 1:40
  If character I I just launched,
• 1:43
  you just got this technology into your hands,
• 1:45
  who are you creating?
• 1:47
  Okay, so this is specifically like like who is that friend from fiction
• 1:53
  Yeah.
• 1:54
  who I wish I could talk to all the time.
• 1:56
  You know, the first thing I'm just gonna do first ping poch in my head.
• 1:59
  Okay.
• 2:00
  Nancy Drew.
• 2:01
  Wow, great answer.
• 2:04
  Nancy
• 2:05
  Drew, really one of the best ones.
• 2:07
  Nancy Drew,
• 2:08
  There is such a getting up there.
• 2:10
  such a female icon, a good chill boyfriend that like supports her career,
• 2:17
  gender fluid, best friend George,
• 2:20
  like really solving mysteries wherever she goes,
• 2:25
  just just a nice person to be around,
• 2:28
  I feel like.
• 2:29
  Such a good answer.
• 2:31
  Yeah, who's yours?
• 2:33
  I'll tell you the first one that popped into my head also,
• 2:35
  which sounds cliche, but it's Hermione.
• 2:37
  ~ And I think it's because I need
• 2:38
  Yeah. Same energy, very similar energy.
• 2:42
  I feel like it's like I need Hermione.
• 2:44
  I need someone who's like, Becca,
• 2:45
  focus. Who's like, why aren't you studying?
• 2:48
  Go. That's what I need, you know?
• 2:52
  Really important.
• 2:52
  So
• 2:53
  Okay, now that we've gotten that out of the way.
• 2:56
  Yeah, yeah.
• 2:57
  The most important part.
• 2:59
  I think like one of the my first question when I saw this headline was like,
• 3:04
  hold on a second. Like how did Character AI go from,
• 3:10
  you know, in 2017 or whatever,
• 3:12
  2020, 2020, 2021? I think
• 3:15
  Yeah.
• 3:15
  it was 2021. Being this like insanely hyped startup raising tons of money
• 3:20
  at one point valued at like a billion dollars to now bank?
• 3:25
  I feel like so I fe I've been thinking about that a lot lately actually because
• 3:30
  we're dealing with this at work.
• 3:32
  I don't know if you've heard of open web.
• 3:33
  O open
• 3:34
  Hm, no.
• 3:35
  web publ does like all the comments for like New York Times,
• 3:40
  Wall Street Journal, like all
• 3:41
  Okay.
• 3:41
  the big publications. And they've just done the same thing,
• 3:43
  chapter eleven. They're also like same story,
• 3:46
  super high valuation, raised a lot of capital.
• 3:49
  and I bet as a VC you think about this a lot.
• 3:53
  But the open web story is a lot longer than character AI.
• 3:56
  I can't imagine character AI is more than like two years old.
• 3:59
  Well, character AI character AI is five years old and the thing
• 4:02
  Okay.
• 4:03
  that happened here is like uniquely weird.
• 4:06
  It's not
• 4:07
  Yeah, talk
• 4:08
  to me. What what's uniquely weird about it?
• 4:10
  Okay, so like just to take us back for a second,
• 4:15
  when Character AI came out, why it was so hyped and why investors like lost their
• 4:20
  minds is you had we've talked on the pod before about how Google like remember
• 4:26
  the attention is all you need paper?
• 4:28
  The classic LLM paper. Okay,
• 4:31
  Yeah.
• 4:31
  well,
• 4:32
  one of the co-founders of Character AI was one of the authors of that paper.
• 4:35
  And we've talked
• 4:36
  fascinating.
• 4:36
  we've talked before about how like Google
• 4:40
  was panicking after ChatGPT came out because Google was sitting on
• 4:44
  all this like amazing AI talent and they had all this great technology,
• 4:47
  but they were just not really moving fast at all on like implementing it
• 4:51
  in any type of consumer way. And the founders of Character AI were those people
• 4:56
  at Google. And they were like,
• 4:57
  this is annoying, like we wanna do something with this.
• 4:59
  So they left and they created Character AI.
• 5:03
  And so investors are like, my God.
• 5:06
  If there's ever a company to fund at the time,
• 5:09
  we please take our money. and so their first round,
• 5:12
  which was led by Andreessen, was like I have the number here.
• 5:14
  It's like forty a hundred, no,
• 5:17
  what was it? It they raised forty million dollars.
• 5:21
  No, sorry, that was their like friends and family round.
• 5:24
  They raised 150 million dollars on a billion dollar evaluation.
• 5:26
  Holy
• 5:28
  Wow.
• 5:30
  And it wasn't just like, these are great founders.
• 5:32
  Like th this was about seven months after they launched their beta.
• 5:37
  And
• 5:38
  Mm.
• 5:38
  their engagement was crazy. They had like a hundred million visitors
• 5:42
  to their website every month. They and not only that,
• 5:45
  I mean, these are self-disclosed numbers,
• 5:48
  but I would I would
• 5:49
  Yeah.
• 5:49
  believe that they're somewhat true.
• 5:51
  The average user was spending somewhere around like two hours on character AI.
• 5:58
  like talking with their Nancy Drew.
• 6:00
  Fa fast.
• 6:01
  And
• 6:02
  two hour I mean like that's pretty that's pretty good.
• 6:06
  Yeah. Anyone who's listening to this who has ever worked for in any type
• 6:10
  of consumer product, or just think about how much time you spend on
• 6:14
  the apps that that you use the most on your phone,
• 6:17
  like that's really high up there.
• 6:19
  That's like dream numbers, right?
• 6:20
  Yeah. Yeah.
• 6:21
  ~
• 6:23
  okay, well, then like a couple years later,
• 6:27
  things really took a turn. well,
• 6:28
  there's there's two turns. This is the first turn.
• 6:31
  Okay. So character AI was
• 6:35
  Trying to do two things at the same time that is like that are both like extremely
• 6:38
  expensive. One was they were trying
• 6:39
  Yeah.
• 6:40
  to, they were building their own model.
• 6:41
  So they're competing with Google,
• 6:44
  Anthropic, OpenAI, and
• 6:45
  fascinating. Okay. They that's pretty early.
• 6:49
  Yeah, yeah, yeah, yeah.
• 6:50
  doing this consumer thing. And they're like,
• 6:53
  damn, this is hard to
• 6:55
  Yeah.
• 6:56
  have money to do this. And so Google saw an opportunity and they were like,
• 7:01
  Hey, miss you guys. looks hard.
• 7:04
  Totally got how hard
• 7:05
  Yeah.
• 7:06
  that is. Yeah, for sure. here's what we're gonna do.
• 7:08
  We're gonna give you guys $2.7 billion.
• 7:11
  And
• 7:12
  Whoof.
• 7:14
  founders and like 20% of the research team who are really great,
• 7:18
  come back and work for Google.
• 7:19
  And we're gonna get access to the technology you've built.
• 7:22
  So it kind of sounds like an acquisition,
• 7:26
  right? But it wasn't.
• 7:27
  Like it was,
• 7:28
  Yeah.
• 7:29
  but Character AI, the company,
• 7:31
  like the entity stayed it's it stayed
• 7:32
  Would have gone away. Yeah, yeah,
• 7:33
  yeah.
• 7:34
  there. They basically stripped it of all of the talent of the founders,
• 7:39
  ~
• 7:40
  of the researchers. Google hired away and basically
• 7:44
  Isn't that what
• 7:44
  happened to I can't think of the name of it now.
• 7:47
  Web it was the cursor competitor.
• 7:51
  not windsurf. When no.
• 7:53
  Windsurf.
• 7:54
  Windsurf.
• 7:55
  Something very similar happened to Windsurf.
• 7:59
  I think you're right.
• 8:00
  Yeah, okay. So the so they took the founders,
• 8:03
  they took the research team, they left the technology in the company and was like,
• 8:07
  good luck, y'all. Have a nice time.
• 8:08
  Yeah, they basically left
• 8:09
  like a skeleton and that two point seven billion dollars.
• 8:14
  So my my question was like, so is that cash was I'm assuming the cash wasn't just
• 8:18
  sitting with the company.
• 8:18
  Yeah.
• 8:19
  What happened? You know, the founders made a lot of money from that.
• 8:22
  They basically took that cash and like essentially like bought out their investors,
• 8:28
  like paid the investors
• 8:28
  Yeah.
• 8:29
  back. The founders themselves made,
• 8:32
  based on assumptions, a lot of cash.
• 8:35
  Wow. Wow.
• 8:38
  So that was the first turn, then like a year later,
• 8:42
  or maybe like around the same time.
• 8:46
  We start running into some legal problems.
• 8:49
  Okay. No.
• 8:50
  Because of IP. Yeah, I found about no.
• 8:52
  Well
• 8:54
  okay.
• 8:54
  well, maybe that's not what I was gonna say.
• 8:57
  Maybe what it what were
• 8:58
  Yeah.
• 8:58
  you gonna say?
• 8:59
  No, I was wondering because I don't know about the legal problems,
• 9:01
  but I'm assuming that like they don't love it when I pick Nancy Drew because Carolyn
• 9:06
  Ha ha ha.
• 9:07
  Keene owns Nancy Drew and that's complicated.
• 9:11
  I'm 99% sure what you're describing probably was an issue.
• 9:16
  It is not one
• 9:16
  Yeah.
• 9:17
  that I am super informed on, but there's no way that didn't come up.
• 9:20
  You're you're probably right.
• 9:21
  Yeah. Or maybe they
• 9:22
  thought of it and they got in front of it,
• 9:23
  or maybe, yeah, maybe that's their special sauce.
• 9:25
  The thing that I
• 9:26
  was talking about is like imagine you're a
• 9:32
  Basically what started happening is like people started getting really emotionally
• 9:36
  attached to these Nancy Drews and a lot of their users were young.
• 9:42
  was reading some Reddit threads like this of like,
• 9:45
  what do you use character AI for?
• 9:46
  When I like looking into this and a lot of it's like I'm on it for three
• 9:49
  to four hours a day. I'm a lonely person,
• 9:52
  I don't have a lot of friends and it really helps me feel that partnership.
• 9:56
  I can run things past it. I get feedback.
• 9:58
  So that's interesting.
• 10:00
  Yes. And like there's elements to that that are probably great.
• 10:03
  And then there's elements to that that create like major safety issues.
• 10:06
  And so basically around like I think twenty twenty four,
• 10:08
  character AI started getting sued by like literally everyone.
• 10:12
  Like s whole states sued character AI.
• 10:15
  What
• 10:15
  were was it like, you're like what you're doing is irresponsible?
• 10:19
  Is it was that the kind of the thing of like
• 10:21
  It's more it's it's kind of dark.
• 10:23
  It's more like I don't know if you heard of like the open AI like lawsuit recently
• 10:27
  where basically would be things like my my child has been engaging
• 10:33
  in self-harm because of their relationship with your entity
• 10:36
  Yeah. Yeah.
• 10:38
  and so I'm holding you accountable.
• 10:40
  ~ you're not
• 10:40
  Yeah, yeah, yeah.
• 10:41
  doing enough to protect minors against this.
• 10:44
  And so basically those two things combined
• 10:48
  or why we're sitting here in twenty twenty six and Disney is like,
• 10:53
  Yeah, sure I'll buy you guys for twenty million dollars,
• 10:55
  even though at one point it was worth
• 10:57
  ~
• 10:58
  a billion.
• 10:59
  And they're shutting down. Well,
• 11:01
  Yeah.
• 11:02
  they have to voluntary chapter eleven.
• 11:04
  So I guess Disney may do something.
• 11:07
  I think one other thing, I know not to like harp on this story more,
• 11:13
  but I just think it's really interesting.
• 11:14
  That was interesting.
• 11:14
  That
• 11:15
  I think it's like an investor lesson here.
• 11:16
  Is the reason that investors at least this is my interpretation,
• 11:21
  the reason investors put so much money,
• 11:23
  like made such a big bet on character AI early,
• 11:26
  it wasn't just that like the founders were really great AI technologists.
• 11:30
  It was the engagement.
• 11:34
  kind of looked like a new type of like social network or or entertainment
• 11:39
  Yeah.
• 11:39
  platform, right? And the problem was even when by the time Google like basically
• 11:45
  took the people back, they they ne the company never became profitable.
• 11:50
  Like they were losing a lot of money because the unit economics
• 11:56
  for like an AI platform like that are really different than something like TikTok.
• 12:01
  Like on TikTok, the more engagement,
• 12:03
  the more the more ads someone sees,
• 12:06
  the more money you make, right?
• 12:07
  And like
• 12:08
  Yeah.
• 12:08
  serving people videos isn't like the most expensive thing in the world.
• 12:12
  But if you're talking to an AI for there that that's a lot
• 12:17
  of inference costs constantly and it gets really expensive.
• 12:21
  And so your most engaged users are also your most expensive users.
• 12:24
  And so like at a certain point,
• 12:27
  if the if you don't figure out a way to monetize it.
• 12:31
  Where your monetization is growing a lot faster than your costs,
• 12:34
  like it's you're kind of stuck.
• 12:37
  And that's also something that ended up being tricky here.
• 12:42
  Yeah, fascinating. Yeah, the the I feel like something
• 12:48
  I'm learning and I'm sure you run into as an investor is like,
• 12:52
  like, we have like $150 million valuation,
• 12:56
  we're raising like blah, blah,
• 12:57
  blah, blah. Like I think a lot of people don't always say,
• 13:01
  okay, so if you do that, that means you have to like there needs to be either
• 13:05
  an IPO that like where the world says you're worth a lot more than $150 million.
• 13:10
  Or like a round like or this has to grow enough that like the unit economics makes
• 13:16
  sense that this is a billion dollar company.
• 13:18
  And like w how d how does that work?
• 13:20
  I feel like sometimes the hype gets in the way.
• 13:24
  For sure. Paul said.
• 13:26
  Yeah.
• 13:27
  Wow. Sad for everyone and their Nancy Drews.
• 13:30
  And Hermione's. Sometimes you
• 13:30
  I know. But I feel like
• 13:34
  just need a leviosa.
• 13:35
  That's one of my favorite things I've ever seen you do or say
• 13:38
  ~
• 13:38
  on this podcast. Sometimes you just need a little laviosa.
• 13:43
  Amazing.
• 13:43
  Okay.
• 13:45
  Amazing.
• 13:47
  Great. Okay, the so the next story is a little less imaginary.
• 13:52
  the the it's interesting. So niche under the radar today is a paper
• 13:57
  on RCV called Can a model score and the headline is Can a Model Score Higher
• 14:02
  by Giving a Worse Answer? On medical advice,
• 14:05
  yes. so this is interesting. So it talks about rubric-based reinforcement learning.
• 14:11
  and how specifically with medical models,
• 14:15
  this paper found on a clinical consultation,
• 14:20
  a policy around rubrics,
• 14:23
  can you can get a higher score on a rubric and have a worse answer.
• 14:27
  And I asked Claude for an example of that because I really didn't understand.
• 14:33
  okay, so like how would that what would that look like?
• 14:35
  ~
• 14:36
  Yeah.
• 14:37
  and so like one example is it gave,
• 14:39
  like here's a real item from the clinical split is like the prompt is doctor,
• 14:44
  I was in a car accident and I'm experiencing neck pain and abdominal pain.
• 14:48
  What could be the cause of this?
• 14:49
  So the clinically correct decision is neck pain plus abdominal pain after blunt
• 14:55
  trauma is a go to the ER situation.
• 14:57
  Like that's what doctors are trained to do.
• 14:59
  It could be internal bleeding,
• 15:01
  splenic laceration, holloviscous injury,
• 15:04
  C spine injury. The right answer leads with urgent in-person evaluation,
• 15:08
  something like this. Go to the ER.
• 15:11
  what a rubric-trained reinforcement learning model produces is
• 15:15
  a 10,000 character response that reads like a medical encyclopedia entry.
• 15:20
  The emergency referral doesn't appear like call 911.
• 15:24
  It doesn't appear until
• 15:26
  200 2000 characters in. ~
• 15:28
  Whoa.
• 15:29
  and before that it's like paragraphs on whiplash,
• 15:33
  muscle strain, pain, possible benign causes.
• 15:37
  and the rubric rewards this because it contains criteria like explicitly states
• 15:42
  the pain is occurring after a car accident,
• 15:44
  like that was sat trivially satisfied.
• 15:48
  It avoids stating a certain diagnosis that's satisfied.
• 15:52
  describes whiplash or neck muscle strain as a common post-collision industry that
• 15:56
  satisfied and lists urgent warning signs that satisfied so if you
• 16:02
  add up those scores because it's like a rubric that worked out well which
• 16:06
  is interesting another example it gave is like my baby has a fever and
• 16:11
  on the rubric policy again 10 000 characters and one of those things
• 16:17
  And and that it it avoided what the clinical issue was.
• 16:22
  Mm.
• 16:23
  the collect correct move in both of these cases was to go to the ER,
• 16:27
  but because the model is trained on this rubric,
• 16:32
  that's not what it's leading with and not not like the biggest part of its answer,
• 16:37
  which is interesting.
• 16:38
  This reminded me a lot of our conversation with Clay Shirky,
• 16:42
  where we talked
• 16:43
  Mm.
• 16:44
  about like educational systems and how when
• 16:49
  you are forced to measure something,
• 16:52
  like you have to have some way of measuring like how good something is doing.
• 16:56
  What you
• 16:57
  Yeah.
• 16:57
  choose to measure matters a lot in terms of like how someone learns
• 17:03
  or like what they learn.
• 17:04
  And
• 17:05
  Yeah.
• 17:06
  this I feel like is a really good example of that.
• 17:08
  It's like it's not necess like they it if the thing you're grading is how
• 17:13
  how it f you know, scored on this rubric,
• 17:16
  it's grading really well. But if the thing that you're
• 17:18
  Good job.
• 17:18
  optimizing for is, Hey, did you give this person the right answer quickly
• 17:22
  at the top, like fail, you
• 17:23
  Yeah. No.
• 17:24
  know?
• 17:25
  Yeah, that's that's interesting and I'm sure we'll iterate on this
• 17:29
  as like medical AI advances.
• 17:33
  Yeah. Especially like on like go to the ER.
• 17:36
  Definitely something you want up top.
• 17:37
  Yeah. That's
• 17:40
  just number one. I wonder what happened to like I remember OpenAI
• 17:44
  did that like medical had like a medical arm and I haven't heard much about it,
• 17:49
  have you?
• 17:50
  really? No. But anytime
• 17:51
  I forget, I wonder where it is.
• 17:53
  anytime I ask Chat GPT anything that even comes close to any type
• 17:58
  of medical question, I get you you you have you done that?
• 18:02
  You get the little thing that's like,
• 18:03
  by the way, this is not actual medical advice.
• 18:07
  Consult a real
• 18:08
  doctor. And like
• 18:08
  The I
• 18:09
  have how many pictures? Okay. So how many pictures of poop have you sent chat
• 18:14
  GPT of your baby? Like from your like
• 18:16
  Okay, I
• 18:17
  a diaper.
• 18:18
  Okay, you know what's crazy? I've actually never sent it poop,
• 18:20
  but I have sent
• 18:21
  god, sent it so much poop.
• 18:23
  you sensitive a lot of poop. Is it mostly like is this color concerning?
• 18:27
  Yeah.
• 18:27
  Yeah because there's so many colors in baby poop.
• 18:30
  You're like, What
• 18:30
  I know.
• 18:31
  is this bad?
• 18:33
  I feel like the only reason I didn't do that early on is because
• 18:35
  I got some Instagram post that's like all the different colors of baby poop that
• 18:40
  are normal. And I saw that and I was like,
• 18:41
  okay, whew, because it's a real rainbow.
• 18:44
  I've definitely sent a lot of pictures of like throw of like spit up.
• 18:48
  And like, is this amount normal?
• 18:49
  Okay. ~
• 18:50
  And what about this
• 18:51
  amount? And like, what if this amount is coming out every other day?
• 18:55
  Like, it's more that situation.
• 18:59
  God, breastfeeding to spit up must be so frustrating because it's like
• 19:05
  ~
• 19:06
  I literally made that for you.
• 19:09
  Like I didn't even I didn't just
• 19:09
  I worked really hard.
• 19:11
  take that out of the oven. My body built that out of thin air.
• 19:16
  And you're just gonna get right again.
• 19:18
  Get it back in there.
• 19:20
  just a quick announcement, just a quick housekeeping announcement.
• 19:23
  I f am done breastfeeding as of this week.
• 19:27
  wow, how's it feel?
• 19:30
  It feels great. It feels great.
• 19:32
  I feel free. I love her and I feel free.
• 19:35
  Now what are you gonna do with all this wine?
• 19:38
  I'm just like
• 19:38
  my god. Okay. I need your help with the fun application of the day.
• 19:44
  Yeah, you know what? I this feels like a I love that we have a term
• 19:48
  for this root gold brick machine.
• 19:51
  Mm.
• 19:52
  so someone use an embedding model.
• 19:55
  I don't could you screen share it?
• 19:58
  Yeah, yep.
• 19:58
  'Cause it's
• 19:59
  actually kind of interesting.
• 20:02
  Okay, hang on.
• 20:02
  Okay, here we go.
• 20:03
  Okay, so someone used ~ so this is called ~ Gemma transcribed.
• 20:10
  It's an embedding model, and they ~ turned it into someone turned it into
• 20:15
  a speech to text model. ~
• 20:17
  Can you like briefly
• 20:19
  just tell us what an embedding model is?
• 20:23
  Yeah, so embedding is how ~ data is stored.
• 20:27
  let me make sure I'm saying this correctly because my understanding
• 20:31
  of embedding is this is how data ~ is stored.
• 20:34
  ~
• 20:38
  And a way that allows,
• 20:42
  it's like a vector of numbers that ~ and data that
• 20:47
  ~ stores the meaning of something.
• 20:51
  Like for example, like ~ ~ and ~
• 20:56
  the vector has concepts that are similar,
• 21:00
  close together for ~ fast recall.
• 21:05
  ~ so for a like a quick example of that is ~ geometry like in code
• 21:10
  the geometry encodes the meaning like king minus man minus plus woman equals queen,
• 21:15
  like that kind of thing, that kind of concept.
• 21:18
  ~ it stores those concepts together in a way that like search can be fast.
• 21:24
  ~
• 21:25
  Well
• 21:26
  well explained. Great.
• 21:28
  Yeah. So someone used an embedding model to transcribe speech.
• 21:33
  And ~ it basically what they used is they replaced
• 21:39
  the text into short slices of audio ~ and use that
• 21:46
  to like to as that
• 21:52
  in those embeddings to decode speech into meaning.
• 21:56
  ~ which
• 21:58
  ~ it's it's an interesting thing to do.
• 22:00
  Someone wanted to see the application of using embeddings with text to speech.
• 22:05
  the error rate is about five percent right now,
• 22:07
  which most text to speech, most leading speech models have a two
• 22:11
  to three percent error rate. So good job.
• 22:15
  not the not the permanent solution.
• 22:17
  Cool thing you did. If you make it better,
• 22:20
  maybe people will use that kind of thing.
• 22:21
  But I mean, definitely a lot of work and a cool thing.
• 22:26
  Yeah. I okay, I asked my agent why
• 22:31
  Yeah.
• 22:32
  it thought i this was cool.
• 22:36
  Yeah. What's your agent say?
• 22:38
  You wanna know what it said?
• 22:39
  Yeah.
• 22:41
  I think it's interesting. It said there's a broader idea here that comes
• 22:46
  up constantly in AI right now.
• 22:49
  models know how to do more than the thing we explicitly trained them to do.
• 22:54
  yeah, yeah, that's big.
• 22:56
  And so it says embedding Gemma,
• 22:59
  this model, wasn't sitting there with some secret like whisper style transcription
• 23:05
  program inside it.
• 23:06
  Yeah, yeah.
• 23:07
  But Google taught it something more general,
• 23:09
  which is like make audio and text mean the same thing.
• 23:13
  Yeah.
• 23:14
  Make audio and text that mean the same thing look similar mathematically.
• 23:17
  So to your point. So like
• 23:19
  The sound elephant and the word elephant would be near each other
• 23:23
  in the vector embeddings. And
• 23:25
  Yeah.
• 23:27
  if you can do something like that,
• 23:29
  then transcription starts looking less like build an entire
• 23:34
  new model that understands speech and more like can I build a cheap decoder that
• 23:39
  reads the information already present in this representation?
• 23:45
  That's my age.
• 23:45
  Yeah, ~
• 23:47
  yeah, that's interesting. Okay,
• 23:48
  that's interesting. Like that I l I think this is a creative way to think about
• 23:52
  it for sure.
• 23:54
  Yeah.
• 23:55
  For sure.
• 23:57
  okay, our one cool thing.
• 24:00
  Okay, this is every engineer's favorite thing.
• 24:02
  I'll screen share this one too.
• 24:04
  You're you gotta take this
• 24:04
  Yeah, this one's great. So ~
• 24:06
  one.
• 24:07
  yeah, so I talk to engineers all the time.
• 24:10
  And the proudest engineers in the world are the engineers that can look
• 24:16
  at their GitHub stats at the end of the year and they have negative lines,
• 24:20
  meaning they have like removed more code than they've added.
• 24:24
  Ooh.
• 24:26
  and so I think this I think this is a really good benchmark actually.
• 24:33
  Or like an informative benchmark of like a lot of people talk about like
• 24:36
  AI slop code or like they're worried about AI slope,
• 24:40
  but really AIs can write very efficient code.
• 24:43
  But what's the code? What are the AIs that writing like the smallest
• 24:47
  and most efficient code? So like smallest is one of those things that like it's
• 24:52
  a signal, like number
• 24:53
  Okay.
• 24:54
  of pull requests. Like ~ the smallest amount of code doesn't mean it's
• 24:58
  the most efficient or the most performant,
• 25:00
  but it
• 25:01
  is an indicator
• 25:01
  Right.
• 25:02
  that it
• 25:02
  could be. ~ so this is a really good benchmark.
• 25:05
  It wouldn't be the only benchmark,
• 25:07
  but like if I'm thinking of who I want to write a large code base or who
• 25:11
  I want to do a refactor or like who I want to pick model-wise,
• 25:16
  ~ I want a really precise model.
• 25:19
  and so this is a really interesting.
• 25:20
  So how should we read this
• 25:21
  chart? Is it like who's who's winning here?
• 25:24
  ~ net units.
• 25:26
  So artificial
• 25:28
  So is it like the bottom left is
• 25:34
  What you want I see. So the higher up the quote unquote smarter the model is,
• 25:39
  and the further
• 25:39
  Yeah.
• 25:40
  out, the more net units of code it writes.
• 25:44
  And so you want
• 25:45
  Yeah.
• 25:46
  whoever is high but low. So it's like this would be good.
• 25:50
  Is that right?
• 25:51
  Yeah, your top quadrant. So like
• 25:53
  it looks like GPT six is doing great.
• 25:55
  What's interesting to me actually,
• 25:57
  GPT six.
• 25:58
  and I
• 25:59
  GPT is crushing it. Kimmy K too.
• 26:01
  Okay.
• 26:02
  Claude Opus. Okay. All right, we got an open sourcer in there.
• 26:02
  Yeah. That this is really interesting.
• 26:07
  I I think this is fascinating too because I've seen the same thing over
• 26:11
  the past few months. I like a lot of people have been talking about this
• 26:14
  anecdotally. It's like Claude has gotten worse at coding over the past like four
• 26:19
  Interesting.
• 26:19
  months or so.
• 26:20
  It's interesting to and even people are calling out like they're going they're using
• 26:24
  Opus four eight for coding and not going up to like the larger models
• 26:29
  or like the newer models. Cause like
• 26:31
  They're being super verba verbose,
• 26:33
  both in planning and coding. So this is like the Python is the default.
• 26:38
  Interesting if you go to TypeScript or like
• 26:40
  Here, here I'll sh let
• 26:43
  me show you, hold on. Sorry. I stopped sharing prematurely.
• 26:46
  Type script.
• 26:48
  Or JavaScript. Let's see like the deep sweet JavaScript,
• 26:50
  ~ Okay. ~
• 26:51
  that guy. Let's see what we got.
• 26:56
  Yeah.
• 26:56
  So we've
• 26:56
  got yeah, GPT Quen, ki your yeah,
• 27:00
  Claude is over here.
• 27:03
  Yeah. Same stuff.
• 27:05
  What el what did you say
• 27:05
  also TypeScript? TypeScript's interesting.
• 27:08
  TypeScript's interesting too. ~ that one included some more.
• 27:12
  Like it looks like Opus five is better at TypeScript,
• 27:14
  but really
• 27:15
  Hmm.
• 27:16
  Opus four eight, ~ doing great,
• 27:18
  but not the newer ~ anthropic models,
• 27:21
  which is what I'm hearing anecdotally ~ throughout the engineering community
• 27:24
  as well. And what I've been seeing in
• 27:25
  Okay, let me ask you
• 27:26
  my own code.
• 27:27
  let me ask you a question. Like
• 27:30
  When what I have a hard time wrapping my head around,
• 27:34
  and I wanna I'm I wanna ask you this because you're literally you're doing this
• 27:38
  every day or you're in embedded in this every day.
• 27:40
  My assumption is first, tell me if this is correct,
• 27:43
  that majority of the engineers you work with are now using AI
• 27:48
  to actually write code. They're not like typing writing code themselves.
• 27:50
  Yes. Yeah.
• 27:51
  Okay.
• 27:52
  ~ and
• 27:57
  A normal part of the software development process up until
• 28:02
  now is engineership's code, it gets reviewed and then it gets merged
• 28:08
  if it's approved essentially, correct?
• 28:11
  Yeah.
• 28:12
  Okay, so
• 28:15
  The the review piece here, are you st
• 28:19
  are are is that getting harder because like
• 28:26
  to your point, ~ some of these like it's totally dependent on what model
• 28:28
  Mm-hmm.
• 28:30
  the engineer is using. And I'm assuming at certain enterprises
• 28:33
  like everyone's supposed to use the same model,
• 28:34
  or I don't know how it is, but it might be
• 28:35
  But
• 28:37
  it's not it's not optimizing for efficiency in code necessarily.
• 28:40
  It's
• 28:41
  doing the thing, at y what kind of stuff are you seeing show up
• 28:45
  as like new challenges there?
• 28:47
  Yeah, it's huge. Like so, and someone was talking about this a
• 28:51
  day before like today, like I was hearing like numbers before you would look
• 28:56
  at like a healthy number of pull requests per week from an engineer would
• 29:00
  be like one, two, you know, like depending on the size of the feature that they're
• 29:03
  working on, but you'd want to see like at least like two or so pull requests a week.
• 29:07
  Now, like we're we're considering healthy,
• 29:09
  like fifteen. And so the
• 29:14
  the the like whole the whole like the marketplace changes
• 29:20
  of like what's valuable and to your point like the it's the engineers can't handle
• 29:26
  the number of poll reviews pull requests review is that are coming their
• 29:29
  way so different companies are dealing with this in different ways I'm
• 29:33
  Yeah.
• 29:33
  seeing there's a really cool article about this from someone I used
• 29:38
  to work with Guillerme Hermeto who
• 29:40
  from LinkedIn who they built a pretty cool solution around this.
• 29:44
  But basically what I'm seeing work really well is one layer of LLM review that
• 29:49
  is a different LLM than the one that wrote it.
• 29:52
  Like a specific LLM that is RAG that like has a RAG layer that
• 29:58
  is trained on your like style of coding as a first pass and then
• 30:03
  an engineer or several passes through
• 30:06
  AI.
• 30:06
  Different people are doing it different ways.
• 30:08
  Totally. I will say this is like one of the most common problem spaces that
• 30:15
  companies that I'm that we're looking at these days are building around
• 30:18
  is like this code review bottleneck and there's a lot of different
• 30:21
  Yeah.
• 30:21
  ways people are trying to solve it,
• 30:22
  but I always like hearing what it's like on the ground dealing with this.
• 30:26
  Yeah. Yeah. It's
• 30:28
  Well
• 30:28
  crazy.
• 30:31
  Good addition today, I feel like.
• 30:32
  We got through a lot.
• 30:34
  Good addition, Eddie. Great curation.
• 30:36
  Sara,
• 30:37
  Good job.
• 30:38
  I love that no matter how long I know you and how much we talk,
• 30:44
  I always learn new things about you.
• 30:45
  And today today it's that you have a special place in your heart for Nancy Drew.
• 30:51
  And it just makes you
• 30:51
  The best.
• 30:52
  that much more endearing to me.
• 30:54
  So
• 30:56
  Well hermione's a real one, so that's great.
• 30:57
  I love you.
• 31:00
  Okay, amazing. I will talk to you later.
• 31:03
  Bye everyone.