Latest / Now Shipping: A Mind the Product podcast / Claude Can Pass The Coding Interview. Now What?

Transcript

Claude Can Pass The Coding Interview. Now What? · Now Shipping: A Mind the Product podcast

• 0:00
  SPEAKER_00: Okay, so a few days ago, an engineering manager at Anthropic posted that he's stopped giving coding interviews to the people he hires.
• 0:08
  SPEAKER_00: His reason's pretty short.
• 0:10
  SPEAKER_00: Claude passes the interviews.
• 0:12
  SPEAKER_00: And if Claude can pass them, well, what good are they in determining whether one person is better than another?
• 0:18
  SPEAKER_00: And he makes a really good point.
• 0:20
  SPEAKER_00: Now, this is coming from Felix Riesberg, and he co-maintains Electron for Anthropic.
• 0:26
  SPEAKER_00: That's the framework a whole lot of desktop apps are built on.
• 0:29
  SPEAKER_00: Now, on October 27th, he posted on X quote, the coding interview is dead, at least for my team at Anthropic.
• 0:36
  SPEAKER_00: And then he published an essay that included some advice for junior engineers.
• 0:41
  SPEAKER_00: So what did he actually say?
• 0:43
  SPEAKER_00: Felix shared how he used to give candidates small coding challenges in real code.
• 0:48
  SPEAKER_00: One example he gives is a little React exercise, but he then goes on to say there's no more point in giving these React challenges than there is in quizzing people about assembly.
• 0:58
  SPEAKER_00: Claude's now better at both.
• 1:00
  SPEAKER_00: He describes these coding tests that he would give as things that were a weak but useful predictor of who would do well on his team.
• 1:08
  SPEAKER_00: But then this month, he stopped using them because now they're not even useful.
• 1:13
  SPEAKER_00: So what replaces it when he's evaluating younger engineers?
• 1:17
  SPEAKER_00: Well, he doesn't really know yet.
• 1:18
  SPEAKER_00: He says he hasn't gotten there quite yet.
• 1:20
  SPEAKER_00: For now, he's back to reading resumes and portfolios, but the thing that he is looking for is small prototypes the person's built.
• 1:28
  SPEAKER_00: And in those prototypes, the thing he especially calls out that he's evaluating is things like taste, empathy, and principled thinking.
• 1:37
  SPEAKER_00: Now, really interesting that he didn't say an understanding of advanced AI models or the ability to churn out millions of lines of code in a given sprint.
• 1:46
  SPEAKER_00: And yeah, I'm sure he's looking for people that are technically proficient.
• 1:50
  SPEAKER_00: I suppose that's table stakes now, but the characteristics he's calling out, it almost sums up the job of a product manager, at least the role that we all want to have, right?
• 2:00
  SPEAKER_00: Now, it's not the first time we've heard about this conundrum of sorts when it comes to coding challenges.
• 2:07
  SPEAKER_00: Anthropic has been fighting this fight for a while.
• 2:10
  SPEAKER_00: Um, there was Tristan Hume.
• 2:12
  SPEAKER_00: Now, he leads performance optimization at Anthropic, and he wrote about this back in January too.
• 2:17
  SPEAKER_00: He talked about how in November 2023 the team needed more performance engineers fast.
• 2:23
  SPEAKER_00: So Tristan posted on Twitter and he got more candidates than the normal interview pipeline could even handle.
• 2:29
  SPEAKER_00: And he spent two weeks designing a take-home test.
• 2:32
  SPEAKER_00: He came up with a realistic problem and he asked engineers to optimize code for a simulated chip.
• 2:38
  SPEAKER_00: Now, candidates could use AI on it, it wouldn't be cheating, because the whole point was to see how they worked with AI.
• 2:45
  SPEAKER_00: More than a thousand people took it over 18 months, and it helped hire most of the current team he has.
• 2:50
  SPEAKER_00: The test essentially worked.
• 2:52
  SPEAKER_00: But then Claude got better.
• 2:55
  SPEAKER_00: By May 2025, Claude 3.7 Sonnet was already beating more than half of the candidates when they handed the work to Claude Code.
• 3:02
  SPEAKER_00: A pre-release Claude Opus 4 produced a better optimized solution than almost all humans in the four hours allowed.
• 3:09
  SPEAKER_00: So Tristan redesigned that test.
• 3:12
  SPEAKER_00: He found where Opus 4 started to struggle, made that the new starting point, and then cut the time limit to two hours.
• 3:18
  SPEAKER_00: And that held for a few months.
• 3:20
  SPEAKER_00: Then a pre-release Opus 4.5 came along.
• 3:23
  SPEAKER_00: It ended up clearing the passing bar in under an hour, got stuck, and then once it was told what cycle count was possible, it found a trick and matched the best human score.
• 3:35
  SPEAKER_00: Now, that human had heavily used Claude as well.
• 3:38
  SPEAKER_00: I should point that out.
• 3:40
  SPEAKER_00: But Tristan looked at two different paths from that point.
• 3:43
  SPEAKER_00: He could either ban AI, but that was pretty hard to enforce, and people still need to work well with AI on the job.
• 3:49
  SPEAKER_00: I mean, this is anthropic, right?
• 3:51
  SPEAKER_00: Or he could raise the bar so a candidate had to beat Claude code on its own.
• 3:56
  SPEAKER_00: But humans spend about half the time just reading the problem.
• 3:59
  SPEAKER_00: So the best strategy would end up becoming just watching Claude work and learning from that.
• 4:04
  SPEAKER_00: So he didn't really like either of those paths.
• 4:06
  SPEAKER_00: Instead, he went with a third direction.
• 4:09
  SPEAKER_00: The current test he now uses is a set of puzzles with a tiny, heavily constrained instruction set, far from anything in the model's training data.
• 4:17
  SPEAKER_00: Opus 4.5 actually failed a medium to hard version of the test.
• 4:22
  SPEAKER_00: And one design choice he made I thought was really interesting.
• 4:25
  SPEAKER_00: There's no debugger, no visualizer.
• 4:28
  SPEAKER_00: Candidates have to build their own tools because how someone invests in tooling tells you about their judgment.
• 4:34
  SPEAKER_00: But he gave something up.
• 4:35
  SPEAKER_00: He said the original test worked because it resembled real work.
• 4:39
  SPEAKER_00: This new test works because it simulates novel work.
• 4:42
  SPEAKER_00: And that's really interesting to think about.
• 4:44
  SPEAKER_00: You could build a test that the model can't beat, but the harder you work at that, the further it ends up drifting from the real job.
• 4:52
  SPEAKER_00: According to Tristan, designing an interview that represents the job is now harder than ever.
• 4:57
  SPEAKER_00: Now let's go back to Felix, who we started with.
• 5:00
  SPEAKER_00: Now, the decision he made, it's not like he was announcing some anthropic-wide policy.
• 5:05
  SPEAKER_00: I mean, he's one manager deciding for his own team.
• 5:08
  SPEAKER_00: But going back to his advice, he boiled things down to a few different things.
• 5:13
  SPEAKER_00: First, he says it's important to focus on business impact instead of working the backlog.
• 5:18
  SPEAKER_00: He says, pick one good idea, not five, and build a working prototype over a weekend instead of writing a PRD.
• 5:26
  SPEAKER_00: Now, I love this.
• 5:28
  SPEAKER_00: I love that it starts with business impact.
• 5:30
  SPEAKER_00: I know my friend Matt LeMay, author of Impact First Product Teams, would love that too.
• 5:34
  SPEAKER_00: Because let's face it, this is really important.
• 5:37
  SPEAKER_00: The products we build are meant to create impact for our businesses.
• 5:41
  SPEAKER_00: It's not just about building cool stuff as fast as possible.
• 5:44
  SPEAKER_00: Yeah, speed matters.
• 5:46
  SPEAKER_00: I'll get to that.
• 5:47
  SPEAKER_00: But Felix gives a very specific example.
• 5:50
  SPEAKER_00: There were two relatively new anthropic engineers, Anthony Morris and Alice Zhao.
• 5:55
  SPEAKER_00: They built a graphical interface for Claude Code over a single weekend.
• 5:59
  SPEAKER_00: The first public version shipped eight days later.
• 6:03
  SPEAKER_00: So yes, speed matters, it matters a lot, but it's not the only thing.
• 6:08
  SPEAKER_00: He also makes a point about fresh eyes.
• 6:11
  SPEAKER_00: He said, newcomers don't know why something is supposedly impossible.
• 6:15
  SPEAKER_00: So sometimes they try it anyway.
• 6:17
  SPEAKER_00: And I love this.
• 6:18
  SPEAKER_00: I've been saying it's so important for everybody, even those of us with a decade or two of product experience.
• 6:23
  SPEAKER_00: In fact, maybe it's most important for those of us with a decade or two of product experience to be okay with being a beginner again.
• 6:32
  SPEAKER_00: Sometimes it's what we know that sort of holds us back.
• 6:35
  SPEAKER_00: Now, I know, again, this is coming from an engineering manager at Anthropic.
• 6:40
  SPEAKER_00: And yes, Anthropic sells coding tools.
• 6:43
  SPEAKER_00: So this whole story, right, where Claude is so good at coding that the interview broke, it's a flattering story for Anthropic.
• 6:50
  SPEAKER_00: I do fully realize it.
• 6:52
  SPEAKER_00: But it doesn't make him wrong.
• 6:54
  SPEAKER_00: Maybe it is fair to take everything with a grain of salt here, but I actually think he's onto something.
• 6:59
  SPEAKER_00: Now, there have been some critics to this whole concept of, like, hey, the coding tests, they're completely broken.
• 7:06
  SPEAKER_00: I mean, some engineers have pointed out that nobody's found a better replacement for the 60-minute interview quite yet.
• 7:13
  SPEAKER_00: Others have said, hey, look, anthropic is unusual.
• 7:16
  SPEAKER_00: A lot of traditional companies will keep running algorithm tests for years.
• 7:20
  SPEAKER_00: A few even pointed out the other side of the junior program that early career candidates, they already struggle to get past resume screening because some companies are reluctant to hire them at all, including Anthropic.
• 7:33
  SPEAKER_00: You might remember just a couple weeks ago, the Anthropic Institute published a report on how employment is changing inside of Anthropic.
• 7:41
  SPEAKER_00: And one finding, junior technical hires fell from 27% of the technical hires they made to 9% since early 2025.
• 7:51
  SPEAKER_00: And the junior share of the technical staff dropped from 21% to 14%.
• 7:57
  SPEAKER_00: Now, this essay tells junior engineers that hunger beats tenure, but the company's own numbers that they released, which we reported on not that long ago, they say that they're hiring far few juniors.
• 8:10
  SPEAKER_00: Now, again, it doesn't necessarily mean that the information is conflicting.
• 8:14
  SPEAKER_00: Both can be true.
• 8:16
  SPEAKER_00: It can be harder for junior engineers to get in at Anthropic, and the way in can look more like a well-thought-out, impact-driven prototype that somebody builds rather than a coding test that they pass.
• 8:29
  SPEAKER_00: But if you're a junior person reading Felix's advice, it is important that you know about this trend with junior engineers at Anthropic.
• 8:37
  SPEAKER_00: Code commits per engineer were up five times from six months earlier.
• 8:41
  SPEAKER_00: So the work of engineers is really starting to shift from coding product to overseeing the work to build the product that, of course, AI agents are doing.
• 8:51
  SPEAKER_00: Now, one source from the research that the Anthropic Institute pointed out that we shared a couple weeks back is that the 20% of the work that was strategic is now 90% of the job.
• 9:02
  SPEAKER_00: And if that's really true, you could see why a React exercise stopped telling Felix anything about whether somebody would be a good team member for him or not.
• 9:11
  SPEAKER_00: Now, is Anthropic an outlier?
• 9:14
  SPEAKER_00: Maybe not as much as you'd think.
• 9:16
  SPEAKER_00: CoderPad recently did a survey of about 650 developers and recruiters, and they found that 34% ban AI completely, 46% allowed in some form, 20% decide case by case.
• 9:31
  SPEAKER_00: Now, the company Canva, they require candidates to use AI tools like Copilot, Cursor, and Claude in technical interviews.
• 9:39
  SPEAKER_00: What they saw is the strongest candidates controlled the boundaries of the problem and reviewed the AI's code with a critical eye.
• 9:47
  SPEAKER_00: The weaker candidates struggled to judge what the AI actually handed them.
• 9:51
  SPEAKER_00: So companies today are starting to not necessarily measure whether you can code, but whether you can tell if the code is any good.
• 9:59
  SPEAKER_00: But what about product management specifically?
• 10:02
  SPEAKER_00: Well, lots of companies are doing different things.
• 10:05
  SPEAKER_00: LinkedIn recently renamed its entry-level APM program to Associate Product Builder.
• 10:11
  SPEAKER_00: They remove the resume upload.
• 10:13
  SPEAKER_00: They now ask applicants for a 60-second video of a product that people use, plus answers to six questions.
• 10:21
  SPEAKER_00: Now, that's just one company and an example of how they've evolved.
• 10:24
  SPEAKER_00: I know lots of companies, in fact, probably most companies, are still asking for a resume at the very least, right?
• 10:31
  SPEAKER_00: But now it's no longer the only important piece that companies are looking at.
• 10:34
  SPEAKER_00: They want to see what you can actually build and whether what you've built actually matters.
• 10:40
  SPEAKER_00: Now, there is another potential problem to this whole, you know, the prototype is the new resume uh concept.
• 10:46
  SPEAKER_00: The same AI that broke the coding test also makes a polished prototype very easy to produce, right?
• 10:53
  SPEAKER_00: A slick demo is pretty cheap to build now.
• 10:56
  SPEAKER_00: So what is it really a signal of?
• 10:58
  SPEAKER_00: It shows that someone can build something, but it doesn't necessarily show how they build the right thing, or if they're willing to kill the wrong idea, or how can they actually work with people along the way?
• 11:10
  SPEAKER_00: Also, demos without a scoring rubric, they introduce bias.
• 11:13
  SPEAKER_00: I mean, measuring things like taste and judgment, it's pretty subjective.
• 11:17
  SPEAKER_00: Felix says he's looking for taste, and I believe him, but taste is the hardest thing in the world to score for right now.
• 11:24
  SPEAKER_00: So, what does this all mean for you?
• 11:27
  SPEAKER_00: So think about the case study that you give candidates in the interview process.
• 11:31
  SPEAKER_00: The product sense question, the improve this app prompt, the take home where they write a one-pager.
• 11:37
  SPEAKER_00: I mean, if you'd happily advance a candidate who passed your test using good old Claude or ChatGPT or whatever they want to use, then is that round actually measuring a person anymore?
• 11:48
  SPEAKER_00: Or is it measuring access that they have to a chat bot?
• 11:52
  SPEAKER_00: I mean, all these tests, you've got to ask yourself, what was the point of the test anyway?
• 11:56
  SPEAKER_00: What were you really trying to measure?
• 11:58
  SPEAKER_00: Were you trying to understand whether a person can take a messy problem, make progress, and tell good work from bad?
• 12:04
  SPEAKER_00: Maybe you should take a look at how your test is actually laid out right now.
• 12:08
  SPEAKER_00: And maybe let your candidates use AI, but watch what they do with it.
• 12:12
  SPEAKER_00: Do they take the first answer that they get?
• 12:14
  SPEAKER_00: And is that obvious to you?
• 12:16
  SPEAKER_00: Can you tell that they pushed back on the AI?
• 12:19
  SPEAKER_00: Does their answer seem similar to all the other answers you're getting from people?
• 12:23
  SPEAKER_00: Or do they notice what's missing from the user research?
• 12:26
  SPEAKER_00: Can you find examples of them being different and with good reason?
• 12:30
  SPEAKER_00: That's judgment showing up here.
• 12:32
  SPEAKER_00: And it shows up in how someone reacts, which is harder to fake in a document.
• 12:36
  SPEAKER_00: Now, one format that I keep seeing recommended is what people call an audit interview, where you hand the candidate a flawed piece of AI-generated work and ask them to review it.
• 12:46
  SPEAKER_00: Now, for engineers, it's a repo with a confident bug in it.
• 12:50
  SPEAKER_00: For product people, try to build the equivalent.
• 12:53
  SPEAKER_00: Maybe it's a metrics plan that tracks the wrong thing or a launch plan that skips the one team that has to say yes.
• 13:00
  SPEAKER_00: Watch what they catch, what they miss, how they explain it to you.
• 13:04
  SPEAKER_00: You may end up learning more in 20 minutes than you did from your last three case studies.
• 13:09
  SPEAKER_00: But set a rubric first.
• 13:11
  SPEAKER_00: If you can't write down what a good answer looks like in advance, you could end up hiring the person that sounds the most like you.
• 13:18
  SPEAKER_00: That's the bias I was talking about before, and you've got to watch for that.
• 13:22
  SPEAKER_00: And look at your job postings too.
• 13:24
  SPEAKER_00: I mean, Felix's whole argument is that hunger beats tenure.
• 13:27
  SPEAKER_00: Well, go on and read your own listings.
• 13:29
  SPEAKER_00: Count how many times it asks for five years or 10 years of experience.
• 13:34
  SPEAKER_00: If the thing you actually want is somebody who builds a working version of an idea and has the taste to know whether it's any good, years on a resume might not be a great proxy for that anymore.
• 13:46
  SPEAKER_00: I'm not saying throw out experience altogether.
• 13:49
  SPEAKER_00: Like experienced product managers, they have scarred tissue.
• 13:52
  SPEAKER_00: Scarred tissue could be an indicator of real judgment, but don't overlook that hungry junior either.
• 13:59
  SPEAKER_00: And hey, if a company like Anthropic is hiring fewer juniors, maybe it's a competitive advantage for your team when you're actually doing the hiring.
• 14:08
  SPEAKER_00: So here's one thing I would do this week.
• 14:11
  SPEAKER_00: Pick one exercise from your PM interview loop and run it through Claude Cold.
• 14:17
  SPEAKER_00: No coaching.
• 14:18
  SPEAKER_00: Just sort of read what comes back and then ask yourself if I got this from a person, would I have hired this person?
• 14:25
  SPEAKER_00: If the answer is yes, I think it's time to start changing your exercise right now.
• 14:32
  SPEAKER_00: All right, that's gonna be a wrap on this week's episode.
• 14:35
  SPEAKER_00: If you found this episode valuable, please subscribe, tell a friend, and if you have any suggestions for me, leave a comment below.
• 14:42
  SPEAKER_00: I promise you, I read all of the comments.
• 14:45
  SPEAKER_00: I will even respond to your comments when I see them.
• 14:47
  SPEAKER_00: So anyway, I will see you right here next week.
• 14:50
  SPEAKER_00: Once again, my name is Mike Belsito, and brought to you by the team at Mind the Product, this is now shipping.