Latest / Artificial Developer Intelligence / Anthropic's S-1, Amazon's AI Memo to Engineers, De-Slopped Qwen & Claude Auto-Evals

Transcript

Anthropic's S-1, Amazon's AI Memo to Engineers, De-Slopped Qwen & Claude Auto-Evals · Artificial Developer Intelligence

• Dan (00:00) hi. Shimin (00:00) hello and welcome back to Artificial Developer Intelligence, a weekly conversation show where this week we have two software developers tracking the impact of AI on our profession. My name is Shimin Zhang, and with me today is my co-hostal pieces, read like diary entries. Lasky. Rahul is still out this week. he is continuing on his path. of world domination. I think he's taking over New America soon. Dan, how are we doing? Dan (00:32) New America. I'm okay. I think my personal pieces actually do read like diary entries is kind of funny. Shimin (00:39) As they should. You're not an AI. Dan (00:42) I don't know, we can look at my blog blog after that I've haven't updated in five years. So I think that's the normal way to do it, right? Shimin (00:46) Yeah. We can take a look at your blog later. I think that'll be fun. we and I have some personal blog stuff later too. Blog experiment. so on this week's show, we are going to start off, as always, with the news thread mail where we're gonna talk about buyout offers at Microsoft and how OpenAI is no longer connecting its models to the internet. Dan (01:10) Then despite the fact that three new models came out this week, we're gonna skip model review Shimin (01:16) Mm-hmm. Dan (01:17) and go straight to post-processing, where we're gonna talk about An update from Amazon Engineering Leadership on the role of the developer in the age of AI. Say that five times fast. And Qwen3-4B with the AI writing ticks trained out. I'm pretty excited. Shimin (01:33) Yeah, and then we're gonna go directly to the tool shed where we're gonna talk about Claude's new automating eval and hill climbing skills. Dan (01:41) Hill climbing. Okay. And then finally we will close it out with what's I hope is a fan favorite. I don't know, no one's ever emailed us specifically about this piece. So if you Shimin (01:51) We've been waiting for some of these news items for a long time. Dan (01:54) Yeah. Yeah, true. It's it's been a very, very big week. So pretty exciting, two minutes to midnight. Shimin (02:01) Well, we cover the financial side of the AI industry. Dan (02:03) Well, yeah, but I mean this week it's not just the financials. Yeah. Shimin (02:07) This week is big. Yes. Okay, well, let's get started. this week, my first news item is Microsoft's offer to buy out around 60,000 employees this week. Dan (02:22) Twenty five percent of their workforce. Shimin (02:25) Yes, that's 25% of their workers workforce. And this is Dan (02:27) Insane. Shimin (02:28) Across all divisions from AI Experience, Copilot, Finance, Human Resources, Xbox, despite the recent Xbox or gaming division layoffs, they are still offering additional folks the buyouts. So, yeah, this is from Reuters they have about 223,000 full-time employees as of June. And this is not the first time they've done buyout offers. earlier, Dan (02:51) Mm. Shimin (02:52) they've done they did a run of 9,000 buyout offers. I believe that was earlier this year, too. And they also had a run of layoffs. so in July, Microsoft announced they will cut 48,000 jobs of sorry, 4800 jobs, or about 2.1 of its workforce. is this A sign that indeed AI is producing increased worker product productivity and therefore the big tech companies are trying to cut their workforce? Or is this a sign that Microsoft has so much money already dedicated on their AI build-out capex that they want to reduce cost in order to look better for the market? Dan (03:35) Yeah, I I had read the hacker news comments when this got posted to HN and one of the ones I found really interesting was that someone on there was saying that Microsoft is traditionally known for like warehousing engineers, right? Shimin (03:49) Yes. Dan (03:50) Which we kind of heard a lot about with the Zerp era stuff. And but apparently they are still doing it, according to this commenter, and they have like entire divisions that are just waiting for like a competitor to screw up, basically, so they can take over in his face. So like Power BI was one of the examples they used. Shimin (04:06) Mm, Dan (04:06) so it seems like maybe they're just kind of like saying like mm in it, it doesn't make sense anymore with AI. You know what I mean? Like parking people when you can Shimin (04:13) Interesting. Yeah. Dan (04:14) iterate so quickly. So Shimin (04:17) Yeah, and of course they're not, ex they're not expecting all sixty thousand employees to take that offer, Dan (04:21) Right, to take it, yeah. Shimin (04:22) right? They've already done the previous round. They kinda have an idea of what percentage with what demographics would take the offer. So they probably Exactly. Yeah. So they probably expect that much. Dan (04:28) Well at a company that big they know anyway. There's probably a whole department that's just like employee sentiment. They've got it down to a science. Shimin (04:37) Yeah, and we should also add that the offer went to level seventy and below. So that's around where I think staff, I think seventy starting in the seventies is is your principal engineer and distinguished software engineers. So Dan (04:51) Did somebody's distinguished according to the thing I read? So yeah. Shimin (04:54) Yeah, so it's pretty wide. as someone who's living in the Seattle area, I'm not looking forward to all these additional unemployed software developers showing up in the job market. Dan (05:03) Ha ha ha. Shimin (05:04) but at the same time, you know, part of me feels like maybe this is kind of hopeful in the sense that you have a lot of people who are near retirement who have all this experience and all this stock options over the years who would create new startups. let's move on to our next news item. Dan (05:23) according to TechCrunch and vis-a-vis OpenAI's own blog, as of October, I believe, twenty-third, twenty twenty six, open AI says it will no longer train its AI models on the live internet, citing at quote unquote significant risks. So they on the 23rd, they released a blog post titled Securing Our Frontier Model Training Environment. and the two really kind of key takeaways from that are that they're only allow offline or cached copies of the web along with like an offline search engine they've created. And in the cases where they're doing this whole like agent swarm thing, they've built an internal message board for them to communicate on. and they won't be allowed to use the the open internet. it's still connected to the internet though. I there's there's like a whole bunch of discourse about like, well, should it really just be air gapped if we're gonna be really safe? and so like I mean, these are still like you know, VMs that are running the agent, not necessarily the compute, but the agent, right? and that's the thing they're talking about, is like securing these VMs, you know, they could be running an AWS or whatever. and so what they did to try to make the security better on the VMs, which I thought was kind of interesting, is they've engaged in a pretty extensive red teaming exercise where they basically intentionally gave agents control of the host and then cut them loose within their infra to see how much havoc they could wreak or like how fast they could get out. 'cause there's other, you know, like defense and depth beyond just the individual host that the agent's running on, that they try to prevent them from having network access. So apparently this part is wild. They spent 400 billion tokens, billion with a B on the red teaming exercise. And Shimin (07:12) Wow. Dan (07:13) they got around 250 findings, where a finding could be anything from, it acted this way, to actually a security disclosure in a piece of open source software. So They claimed that they'd sent most of those up to the, parent projects and tried to get it, do disclosures to get it fixed. the other interesting thing is like around a hundred billion tokens, it was pretty much diminishing returns. They'd already found like well over the 250. Yeah, and then just Shimin (07:37) Most of the exploits. Yeah. Dan (07:39) kind of like flattens off, but they kept spending. I guess if you got them, spend them. yeah. So one final piece I wanted to bring that that's you know. A little bit of supposition, I guess, from the the TechCrunch author is a little blurb, but then they say whether that strategy will perform as well as letting models run freely on the open internet remains to be seen. But it's noteworthy that both Frontier Labs have determined their most powerful technology can't be left unsupervised on the digital commons. so like I thought that piece was interesting because I guess a lot of the stuff that they're doing is related to like RL or like training post training runs. So there there could actually be an impact in the next generation of models by choosing to do this. So it'll be neat to see what happens. Shimin (08:20) Yeah, you think the recent PHP to shell incident, the PHP bug that caused everybody to go into DEF CON one was related and slash a part of these two hundred fifty. I know. there's another new phobia key came out today. Yeah, it's crazy. Yeah. Dan (08:30) It was a bad week. Between that and the the four bit key thing for the root CAs. really. Good lord. Shimin (08:41) it turns out cybersecurity really likes the number four. And you know Dan (08:45) Well the number two times two. Shimin (08:48) Yeah, exactly. earlier when we talked about hugging face incident, you know, part of us was like the cynical part of us stated that this seemed like a part of a marketing stunt, right? Like to Dan (08:58) Uh-huh. Shimin (08:58) say to to leave them in a kind of shoddily built sandbox and say, Hey, it started hacking things. But very quickly we saw that that is probably not the case. They started hacking government sites. I I don't think that's the kind of positive advertisement in quotes that a Frontier Lab would like to have. And it seems like they are starting to crack down on these AI agents. the open question for me is if these agents in a misaligned state can find this many bugs and security issues. And it also came out, I believe, last week that some of these sub agents were not given a basically a board to chat with each other. They managed to communicate with each other via the CPU cache which to me, again, that's that's my reaction too, it like wow. if these agents are not properly aligned, how do you unleash them? into the general public. Slash if you have a a hacker who is able to get their hands, either get their hands on a jailbreak or somehow able to remove the alignment, the amount of damage they can cause is I don't want to say unimaginable because I'm imagining and it's not pretty. Dan (10:10) Yeah. Well, hopefully, with all these new guardrails on the training runs, they'll just get dumb. Ha ha. Shimin (10:16) Well, yeah, let's hope they really get the alignment correct. Otherwise, we're quickly finding out whether or not the Frontier Labs are going to be responsible for the hacks of their agents as a part of the services that they render. Okay, let's move on to post processing, where this week I have an article from Amazon. and what it is is a descrip description of a internal memo from the Amazon engineering leadership on how they are tackling the AI transition, the tsunami that is facing the software development industry at large. one thing I want to say is, you know, say what you will about Our personal feelings of AWS and its dashboards, et cetera, et cetera. I've always really liked their internal memos and their and their technical write-ups, right? They're always very high quality, easily readable, So all that said, What does this memo say? this is a internal memo targeted as software engineers and software development managers. the core problem, of course, is how how should a Amazon deal with AI? And they've come to the same conclusion that I think this podcast and everyone else has, which is that AI is changing the work of software engineering. both more quickly and more fundamentally than any previous trends. And this is quotes. and that they don't see this changing soon, that this change is both quick and gonna be durable. So what are they offering as a solution? they Came up with the same fundamental question that I think a lot of engineering organizations is facing, which is how do you measure how well the software teams are using these new AI tools? Right? They quote, believe that it is the best to measure and be measured by long-term outcomes. but These outcomes can be time delayed, difficult to measure, and hard to isolate from other trends. And in some of these cases, they tackle these problems by using input metrics instead, like token usage or adoption of AI tools. Or Dan (12:27) Mm-hmm. Shimin (12:28) throughput metrics like code changes and reviews. we are all aware of, we here as Amazon are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters. So this is the fundamental question, right? Is It's always difficult to evaluate how well a software engineer is actually solving business problems. So we are left with these opaque metrics like how many tokens they're using. and Even the organizational leadership who started using these metrics know that these are deeply, deeply flawed metric, but they do not have a better option. And part of me sympathizes with that. Right. Like we talk a lot about how token maxing is is is an easily gameable metric, but we don't necessarily have a better metric that we can point to that will apply to an entire organization. Dan (13:21) Yeah, but that's also not a new problem in software engineering, right? Because Shimin (13:26) Right. Right. Right. That's true. Dan (13:29) you know, like were pull requests ever a good proxy? No. Were lines of code Shimin (13:34) No. Dan (13:34) ever a good proxy? No. Were number of Jura tickets cleared? No. Like, you know, it's something that people've Shimin (13:41) If there Dan (13:42) struggled with. Shimin (13:43) but if there was a better metric, that better metric I'm sure would have become the industry standard. And we just really haven't had one we would like to say tying our software developer output to a company's revenue will be a great metric to have. But even that is a both short term thinking and and b impossible to calculate in practice. So Dan (14:04) And also not always representative either. because you got things like throwaway examples that are done purely to prove a point or to prototype something or whatever, and does that directly? it's not shipped code that's gonna impact the business value, the business's bottom line, but in a way it does and contributes it to it because that prototype or whatever might have made it caused the, executive to make a decision or something that did impact the bottom line. Shimin (14:32) Right. C credit assignment is just incredibly difficult. How much of the credit do you assign to the DevOps team that makes sure the feature is up versus yeah, exactly. It's impossible. but Dan (14:40) Yeah, it's true, keeps the lights on, yeah. Or the network team that makes sure everybody's got enough IP addresses to run their you know, it's like the talk about unsung heroes, Shimin (14:50) Yeah, Dan (14:50) like Shimin (14:50) it's it's it's incredibly difficult. So the conclusion they came to is that these input metrics correlate well with the output that matter and expect these flawed metrics to do so for some time. So at least in Amazon's view that PRs and token maxing, at least in the short term, would still correlate to business value created, which is why they're continuing to use it for a little bit. Which, you know, is is kind of odd, but I then again I don't have a better answer to the question. The other thing they talked about is the importance again to understand the customer and understand the product. That is one of the things they called out as you know the work that is not changing as opposed to the work that is rapidly changing due to AI. So they talk about as software developers, we should document the edge cases where AI does slash does not do a good job at automating a particular piece of workflow. So and quote here, quote. If your work does not benefit from current AI tooling, that is a finding, not a failing. Document why. Share it with your community of peers, bring it to your manager and work to find solutions. an engineer who can explain where AI does not yet help their work is demonstrating the judgment that they're asking for. unquote. And I I find that to be to be also rather pragmatic. Like we should have a running list of things that AI is not yet good at doing as an organization. Dan (16:22) Or isn't cost effective at doing either too, right? there's some times where it's just fine at doing it, but it's just like okay. Like should I should we really be I mean, bad example, but true is like should we use it to manage our calendars or Yeah. Shimin (16:38) Yeah, or does it yeah, just kill the context by badly managing our calendars. And the last thing, contrary to at least according to this memo, contrary to what I've heard happening in some parts of the industry, Amazon is calling for both managers and software developers to spend the time to learn AI and make time for essentially everyone to fulfill the curiosity and to truly discover what AI is good at. This is contrary to some of the situations I've heard, which is in some organizations, you know, the leadership pushes AI tools down towards the individual developers. And then at the same time, they raise the expectation of those developers. So you're given a new tool and are essentially told, hey, now you can work five times faster and I expect you to work five times faster and not given the developers the time to actually organically discover what how AI can best help their with their individual workflows. that leads to crunch. And I am glad that you know Amazon's calling that out as as necessary to to fully diffuse this technology through the organization. you know, like maybe Amazon isn't one of those like two person dev shops making in a billion dollars in ARR, but I like to think they are taking or from reading this memo, I think they're they're taking a pragmatic stance despite you know whatever misgivings we have about token maxing and using PRs to account for a developer's productivity. Dan (18:17) Yeah. Except that they're still using those benchmarks, but Shimin (18:20) But at the very end, they still talk about the most important thing is to be passionate about your customers and a long term focus on delivering results, which that is something that I think we can both, you know, sign off on. Right. Yeah. So a perspective on how to adopt AI in your organization from a very large company. on to something completely different. Dan, you've been complaining about AI slopwriting. Yes. Dan (18:48) Slop cannon, the slop cannon that's aimed at all of us and the whole internet. Yeah. Shimin (18:54) Constantly. That's that's that's a load bearing problem that we have. Dan (18:58) If it's not load bearing, it's yeah. and that kinda gets to the heart of it, right? Which is that I don't know about you, but I'm finding myself every day exposed to more and more and more and more and more writing done by AIs, prompted by humans. and it it doesn't get any easier for me, at least in my experience. Like reading it, I just I hate the slot markers so much. And I just like I I almost feel myself getting like physically re revulsed when I read that, Shimin (19:31) I will read like a long Claude response and then I will f I will see the words load bearing argument and I would just like Dan (19:38) Yeah. Shimin (19:38) almost flinch a little bit, you know, it's it's that repulsive. Dan (19:42) And so the this article is really cool because I I part of me wants to know, like, okay, if if more or less, and I know this is a gross generalization, so bear with me, audience here, but like if LLMs are essentially trained on the entire corpus of like human writing in general, right? Why are they so bad? Right. And so this article actually kind of answers it in addition to solving it on a on a small scale, which I thought was pretty cool. So it's by Eric. I'm sorry, Eric, I'm gonna butcher your last name. It's Kanike. and pretty cool blog just generally, by the way, it's got a lot of neat stuff on here. But so he did a previous experiment where I think specifically on Qwen34B which is not a huge model. I think it's like two and a half gigs. and four B instruct, that's what it was. And The that type of writing is actually a vector. It's it can be represented as like a vector essentially and can be steered out. so once he figured that out, he was able to essentially write a training method that adapts if I understand it correctly, he basically did like a post-training. So he had adapted a LORA to each of the the layers that essentially persisted that steering. and so based on his own slop scoring mechanism, which is also what he used to do the training. so I think that was like a human chosen set of like big no-no words like we were just joking about, right? Or big no no things like lists of three Shimin (21:17) Mm-hmm. Dan (21:19) things, you know, or like skipping the transition words that humans would use. he was able to basically score the original model and then several iterations on this until he was able to reduce that score Shimin (21:35) I think it went from one twenty one to twelve. Yeah. Yep. Dan (21:35) twelve or something. Yeah. Yeah, 12.5. So it's not perfect, but it reads a lot more human. And I can quickly read you the one of the examples just to sort of give you a sense of it if you're if you're listening. So the prompt is write a LinkedIn post about a company switching to a new CRM. So the original Quinn 34B instruct. We're excited to announce a major step forward in how we manage customer relationships. After months of evaluation, research, and team input, we're officially migrating to a new CRM platform, CRM platform name, to empower our sales, marketing, and customer success teams with smarter tools, better visibility, and a more seamless customer experience. Why the change? Real-time data access across teams, deeper customer insights with AI-driven recommendations, streamlined workflows to reduce manual tasks. Improve collaboration and automation, a more intuitive, user-friendly interface. so then here's what the deep D slot model said for the same prompt. And it's definitely not like less excited about it. Like, I don't know if you'd necessarily want to post this to LinkedIn, but like it is it reads so much more human. It's kind of wild. So here it goes. We switched to Salesforce last month and it's already changed how we handle customer calls. Before we used a spreadsheet that got updated by different people at different times and the data was messy. Now every interaction is logged in the system and we can see the full history of a customer's engagement. That helped us spot patterns and complaints and adjust our support process. The sales team noticed they could track follow-ups more easily and didn't miss deadlines as often. We still have some training issues, especially with the mobile app, but the team has been working through them. The IT department said setup took about two weeks and the transition was smoother than expected, blah blah blah, goes on and vein as you can see it's like drastically different the the other thing I found really fascinating is the the reasoning of of how he believes that this happens in models because I've long wondered that so in the other the the post where he sort of discovered that vector thing it goes into some detail about it and his hypothesis is that the Language exists in the pre-training phase because it picks up marketing language from the internet. So that's what all of this is, right? You can think about ad copy, it's always very punchy and like excited and Shimin (23:49) Yeah, I I do want to say one of the things that is different between the two versions is the second version, while it reads a lot more like human written, it doesn't read like something someone would post on LinkedIn. Right. Right. Dan (24:00) Right, exactly. Because it's not marketing ready, right? know, so it's like it but it's interesting. So like that comes from the pre training, but then I guess apparently, according to him, it's like RL or R L H F H F that's causing it to come out that way because humans like how it sounds, particularly in assistant tasks, and so they've rated it highly. And that's why it's why they're starting to write that way, is because people The whoever the raters are are like, Yeah, that's great. Sounds good. Ship it. Shimin (24:30) Right. it goes back to what we were talking about last week. is are there neurons that are monosemantic? Right. Is there a single neuron that actually just handles LinkedIn speak? Or I he Dan (24:39) It's the marketing neuron. Yeah. Shimin (24:42) he claims to have found a neuron that only handles it's not X, it's Y. And that's one of the neurons that he patched. Dan (24:50) Mm-hmm. Shimin (24:51) it'll be kind of amazing if if we can reproduce that and take a look. Yeah. Dan (24:55) It would be cool, especially since I would absolutely love to have a halfway decent writing model, so yeah, so the other neat part is, you I'm a big fan of openweight stuff. So you can go ahead and try this out. It's on Hugging Face. and we'll, you know, link this post in the show notes. So if you're curious, and it's only a two point five gig model, so you could actually run it on pretty modest hardware, even if you don't have a fancy, fancy rig. And if you do, let us know what you think. Shimin (25:21) Yeah, and the last thing I wanted to point out too here is in terms of their slop score, the score of these slop habits that he found, the original model is at 121. Just by prompting for a more plain style system prompt, he was able to get it down to 61. And then prompt plus directional steering got it down further to 14.5. so even without the steering, just simply prompting and like prompting with a little bit of additional steer, Dan (25:50) Nudge. Yeah. Shimin (25:51) yeah, is can get you most of the way there. Right. You don't need to go through that st the distilling part necessarily. Dan (25:57) Yeah. Maybe I'll have to play around with that too, 'cause I've got the steering stuff on D S four that I've been starting to play with. so be interesting to see. Shimin (26:05) The hard part is discovering that steering vector. but once you discovered it, it it will be Dan (26:09) Right. Yeah. so Shimin (26:12) you can also greatly increase the steering direction and see what what happens then. What is like you know, Dan (26:18) god. Shimin (26:19) marketing speak over 9,000 looks like. that would be really interesting as well. Dan (26:22) I don't think I don't think society can handle that one. Shimin (26:25) Okay, on to our tool shed. this week I want to talk about Claude's new eval and hill climbing skills. This is a part of the Claude API skill. let's take a look first at the blog post that that that comes with the eval and hill climbing skill first of all, what do you think when you think of evals, right? Dan, what do you think when you think of evals? Dan (26:52) knowing how good your system is that has ML somewhere in the loop. Shimin (26:56) Right. So Dan (26:57) be it a agent or not, doesn't really matter. anytime you have ML in there, it's like basically how good is your non deterministic part of your system doing. 'Cause it's more like a probability than a you know. But yeah, yeah, yeah, binary Shimin (27:10) Then a deterministic test. Yeah. Dan (27:12) answer kinda. Shimin (27:13) Yeah, so we think of it as what unit tests is for deterministic tests, evals are for these non-deterministic systems, right? Dan (27:20) Okay. Shimin (27:20) and the blog post first pointed out four in my opinion, like really good criteria of a good evaluation design. first one is that the evaluation tasks should mirror production. So like if you're Task is to create a shopping assistant. do not ask it historical questions, right? Like your eval design should be as close to someone looking for a thing on your page as possible. Dan (27:47) Uh-huh. Shimin (27:48) and the next eval design principle is performance should improve with stronger models and more thinking. the cynical person might say, hey. This is because you want the underlying agent to use more powerful models. But I think the good reason and and and correct reason is if what you want from the model is a performance that gets better with stronger reasoning models as opposed to something else like missing data, right? Like if your agent has no access to your catalog, a stronger model will not be able to assist the customer better. Dan (28:23) Right. Shimin (28:24) Because it's fundamentally lacking information. So so you should expect the performance of a sonnet model with your eval to be better than haiku. Otherwise, something is probably incorrect or an exaggeration. Like your Opus should definitely work better than a haiku. Cause otherwise, just use the haiku. Like the thing that you're evaluating Dan (28:41) Everybody hates on haiku. I just Shimin (28:44) for. Yeah, haiku maybe does not, or or your Qwen 3.6 or whatnot. And third principle is there is passable headroom at the frontier. I'm gonna quote directly here the most capable model at the highest average should be well below 100% on evaluation. this is super important this is something I think a lot of software developers miss, right? Because we come from a world where your unit test coverage should be always green. But in the e val land, You should probably aim for 70 to 85 percent. Because if you're getting 100 with the frontier model, what do you do when a better model comes next month? You Dan (29:19) Mm-hmm. Shimin (29:20) have no idea how to evaluate it. And then Dan (29:21) Makes sense. Shimin (29:22) yeah, and then the principle number four is low run-to-run variants. since these are eval models, they're never gonna be a hundred percent. So If you run the same model on an EVL twice, once is 75% and the other time it's 29%, you no longer have this like all unit tests is passing feedback. Right? Because if to you all green means passing around 75% of time, then a high variance EVL set would be like a flaky test that only sometimes passes. Okay. So then what the actual skill does is it would take your code base, it will interview you, take a look at a code base, and gather data sets to create an evaluation set, slash it will create an a judge based on that sample data. So it could take at you know X number of both passing and failing tests, and it could either write regex, which are code-based deterministic evaluations, or it could use AI as a judge, right? L as judge. so you can use it to say, hey, I think this request, the response was two or both, now with duck points based on it. And once it has built your evaluation harness, What is really cool is then now we've turned it into basically an RL problem, right? This is the quote, make my code faster meme comes in. Now you can hill climb your existing workflow with the help of this evaluation harness to make it better, right? so if you have these examples where For example, you're using it's not X, it's Y too often, or you're using M-dash too often, then your hill climbing agent can iterate on the prompt and say stuff like, never use dash, and then run it on your set of email tasks to see, well, okay, how many dashes actually came out? which is pretty cool because once you have an eval set. You can then use that as a target. Now we're gonna go to a second part of this tool shed, which is a blog post from Hamel Husain who is a ML engineer who worked at Airbnb and GitHub, and he also does a series of course on evaluations on Maven. so he life checked out this new evaluation tooling and his feedbacks are, you know, it's let's start with the good, unlike the blog post. Let's start with the good here. The good Dan (32:00) Yeah. Shimin (32:01) is that he found the evaluation to be actually pretty good out of the box. It found a lot of issues that other auto-eval approaches are not able to find. so it's it seems like a really powerful just baseline to work with. And now let's go to the bad. the first thing he found that this is not great at is it asks you to create an email before even looking at the data. So it would in Hamill's case, he wants the driver, the the human, to really first go through samples of the AI workflow working and not working to get like a good understanding. Of hey, this is where the workflow is falling short. Whereas the auto-eval approach, of course, it would just find the data and and quickly build an eval set without getting that understanding first. So he didn't like that. he also really didn't like the terminal-based approach for flagging which evaluation should be the correct output. This makes sense. It's it's hard to do data discovery without some a complex UI system. And Dan (33:04) Mm-hmm. Shimin (33:05) if the alternative is you have an Excel sheet and like at column Y, you just put in the your personal score for a particular eval task like that, and then go back to your terminal to give it, you know, your grades. That's that's not a very good evaluation setup. It's not a very good UX. And then Thirdly, he found that the evaluator scope was too broad, that it was trying to do too many things at once. I find that to be like where where does the human taste and judgment come in? Well, here, here is the taste and the judgment, like how broad the evaluation set is and what it shouldn't should not be testing. that is really up to the humans and should not be Completely automated. So as far as a automated evaluation generation set goes, it seems to be pretty good. So of course we can't just do a second hand result of how this evaluation is. we should also do some independent research. So what I have is Dan (34:03) Yeah. Shimin (34:04) I have I have this Dan (34:05) You didn't say this is a vibe and tell you tricked me. Shimin (34:09) This is both a vibe and tell and a tool shed. Dan (34:12) Vibe Shed Shimin (34:12) By yeah, by vibing in the tool shed. I have a skill for Claude to write like me that was scraped from some of our podcast transcripts from a long time ago. And I've never really used it very much. So I wanted to see if we can use the eval skill to iteratively refine the skill to make it actually be closer to my Actual writing, right? so first we need to take a look at what does it think my writing is like? for the record, I fed it a whole bunch of my old blog posts and it calculated some of my writing ticks and how the V1, V1 is the original Write Like Shimin Skill Dan (34:56) Mm-hmm. Shimin (34:56) which was trained on podcast transcripts. in the in my actual writing, I write I think 0.3 times per a hundred sentences. Whereas on the podcast transcript, I say I think a lot more. So it is four standard deviations above. Dan (35:13) Wow. I wonder what that says about Shimin (35:15) Yeah. similarly Dan (35:17) me and Rahul. Shimin (35:18) We can also do this on your podcast transcripts. Dan (35:20) gosh. Shimin (35:21) it also look for other and these are just entirely programmatic regex based searches. So I look for stuff like hedges in sentences, sentences under eight words, and sentences opening with so and but. these are all basically my visual speaking ticks that the original skill catches on, but doesn't actually mimic my actual writing. Right. Dan (35:45) Mm-hmm. Shimin (35:47) similarly it also look for generic Claude code ticks, like it's not X, it's Y per 1000 words. Dan (35:55) What are you talking about? I've never heard that one. Shimin (35:58) my frequency is 0.4 in my writing and per Dan (36:00) For one thousand worse. Yeah. Shimin (36:02) 1000 words, yes. And the original V1 has a frequency that is three standard deviations above. Not using any skills, just have Claude do it by itself, also is three Dan (36:15) Three point three, yeah. Shimin (36:16) set three standard deviations above mine. Which not surprising, right? This is what we're talking about with the LM slop. Dan (36:22) It's not surprising. It's confirming. sorry. Shimin (36:24) Exactly. So then it used a batch of my old posts and It ran three iterations of hill climbing by editing the text of the prompt to reduce a lot of these visual text to be sometimes even below my baseline rate. So we're just talking about it's now x, it's y. It went from three standard deviations above to only one standard deviation above. which is quite good. It also tuned down some of the other outliers, stuff like honestly, actually, and sentences, opening with so and but. that actually went down to a negative 1.2 standard deviations. So just kind of doing it automatically seemed to have done a pretty good job. And here I actually have some of the text differences between v1 and v3. so v3 added first it differentiated my speaking voice and my writing voice. and also it included writing samples and my voice profile. talks about how I like to write my posts starting with on a concrete scenario and how long the sentences should be, how often I cite a source by name, my humor, dry and visual. I don't I don't know what visual part means, but I have to ask AI, I guess. and lastly, it also have a list of things that the AI should not use. For example, it's not X, it's Y is in the bad bad list. that AI should not use the word actually very often. and Also these signposted churns like here's the thing, here's a catch, why does this matter? To be clear, should be removed Dan (38:09) So punchy. Shimin (38:10) from the skill The skill got quite bloated, it went up to like 1700 words, which which is a lot. Dan (38:17) Yeah. We didn't need that context window anyway. Shimin (38:22) yeah, I'm not reading it usually anyways. Dan (38:23) Ha ha. Shimin (38:24) So Yeah, the scale that markdown went from three hundred and seventy nine words to seventeen hundred and twelve words, which seems like a lot to me. last Dan (38:33) So did you write a blog post with it? Shimin (38:36) I of course. The last thing I did was I took some of my actual articles from a couple of years ago and had it rewritten as both the original V0 scale and the V3 scale. So the original article starts with: Have you ever come across a piece of clever JavaScript code that causes you to slow down and concentrate a little harder in order to digest it? V1 changed that line into I keep running to deliberately compact JavaScript. One linters that can change ternaries and logic operatings like they are being charged by the character. And every time the advice I get back is some version of favor readability optimized later. These are really short, really choppy sentences that actually isn't Dan (39:19) Mm-hmm. Shimin (39:20) like the way I write. But it probably is similar to how I how I speak. and then V3 after the climb, the blog post starts with I ran into a fizzbuzz one-liner in a code review this week. The kind of compact JavaScript you run into every so often and have to read twice. It worked. It was also the kind of line that makes a reviewer pause and ask the author to spell it out. Not perfect. It Dan (39:44) But it's yeah. Shimin (39:45) still had this like one liner of it worked. That's that's the kind of writing that I find that, you know. Dan (39:48) Mm-hmm. It's a little still too punchy, yeah. Shimin (39:52) Yeah, exactly. And if you kind of just Dan (39:54) But it's probably about as good as Rahul impersonating your voice, Shimin (39:58) It's it's not terrible. No, I think I think it does it does a pretty decent job given it has a pretty small sample size to make that iteration from. So overall, While the e eval scale did show up a lot of my session, I maxed out my five hour session building and testing this thing. I did find it to be pretty good for just like a blind auto, like see what you can do to hill climb. I would try it on a production workflow at least. Just try it. Yeah. So Dan, now it's time to train your own. So your model doesn't speak with AI slop. Dan (40:31) You have to show me how to do it. I don't know if I got all that just from this post, but yeah. Shimin (40:36) Yeah, it's basically two skills. You invoke one after another, and Dan (40:39) And it builds everything for you to do the lower. Shimin (40:42) you need to give it data. You need to give it data of what good and bad looks like, and then it will build the evaluation, the judge based on the difference between the two. So if you have like slop writing and good writing that you like, I think it'll be interesting to try and build like a skill for say Take your favorite podcaster, me. just kidding. or or favorite author, Hemingway, and see if you'll prefer an AI that, you know, outputs text and writing in the style of a particular author. that kind of persona based AI experiments is is really interesting and exciting to me. All right, shall we move on to two minutes to midnight? Dan (41:23) I think so. It's probably the most exciting one ever in the history of this podcast. Shimin (41:27) All right. If you're new to the pod, Two Minutes Midnight is where we talk about the financial side of the AI industry using the Armageddon clock from the Bulletin of Atomic Scientists, where midnight, zero minutes is the bursting of the AI bubble. we are currently at four minutes. Yeah. Then why don't you go first? Dan (41:51) And you're gonna make me lead with the boring article. Shimin (41:53) We can both talk about the not boring article. Dan (41:57) yeah, so from TechCrunch, again, I I swear I don't work for them or anything. I just they've been writing really good AI content lately. we have one about Meta where Meta wants to pay for its broad AI build out with a newly created class of stock. so it makes a good headline, but like what's actually going on there is like not Super exciting. Basically, they've made a they have a class A and B stock currently that has voting rights. They're making a class C stock that does not have voting rights, essentially so they can do a split without diluting Zuck's ability to run make, you know, decisions for the company. Shimin (42:39) Right. And what's really interesting is Zuck always have these special shares of stocks where he maintains the voting rights on. Right. Like he has a disproportionately large ability. Yeah. Dan (42:52) Yeah, it's sixty percent or something. It's wild. yeah, so I think the reason why they're doing it is honestly more interesting than the structure, which is something we've already sort of talked about a lot, which is like, you know, debt raises are going up to fund these build outs. Meta is pretty convinced that they have found product market fit with their Muse agent stuff. I mean it's certainly setting some App Store records, so maybe they're not entirely wrong to be thinking that. but they have to fund data center build out just like everybody else. So instead of reaching for debt is the instrument to do that, they're hoping to do it through equity because they will get more favorable terms on doing so on any future debt that they raise essentially. So yeah, that's what it all comes down to is making money, build them, build them computes. and Shimin (43:40) Yeah, and this is on top of the existing additional bonds they've issued to help with AI build out. So they're looking to get more money essentially. Dan (43:44) Right. Yeah. Yeah. but but their CapEx like capital expenditure forecast for twenty twenty six was a hundred and thirty five billion to hundred and forty five billion. just just just to build the Shimin (43:57) What is money? We haven't said that in a while. What is money? Dan (44:00) just to build data centers. So yeah. Pretty wild. Shimin (44:05) All right. Well, and this one I think we will do together. But and this one actually is a team effort here because Rahul was the person who submitted this particular post. but he's still Dan (44:15) That's true. He's not here in person, but he's still sending us articles. Shimin (44:19) contributing, yes. So Anthropics S one just came out this week and we have a only CFO's newsletters take on the S one, because it was very long and we didn't have a chance to go through it page by page. Dan, what was your top takeaway f from the various things about the S1? Dan (44:41) their the It's crazy how fast they've grown as a company, I think was the part that was pretty shocking to me. it's like I've known it intellectually, but then seeing it on paper is like, wow. You know. It's it's really astonishing. And I and it almost makes me not believe the hype, but like understand why some subsets of people are so hopeful about the ability to maybe pay back these loans, that they're taking to do the build outs. I I Yeah. Shimin (45:10) Right, their revenue increase something like fifteen X year over year. Dan (45:14) I still think that there's a lot of Kool Aid drinking going on, but like I can I can sort of see it seeing those numbers. That's all I'll say. Shimin (45:20) I'll turn you into a believer. well, yes, their top end first half of twenty twenty-six revenue annualized is sixty seven billion with a B dollars, Dan (45:33) Mm-hmm. Shimin (45:34) from poultry amount of I think three like the year before. so the growth has been just unbelievable. my Biggest headline takeaway was that their gross margin was listed at eighty five percent as of the first half of this year. That is raw, yes. Dan (45:52) Well, that's raw though, right? So they're doing some financial tricks to get to that number. Shimin (45:58) gross margin is just how much the customers are paying them minus how much they're paying on computer. Dan (46:04) Right. Shimin (46:05) Right. So for every single API request token, every single dollar in API requests, they only spend fifteen cents on compute cost. Dan (46:15) Mm-hmm. Shimin (46:16) Which is shockingly high. Dan (46:16) But it's Shimin (46:19) Yeah. And the number still looks good once you take R and D into account. So I believe this particular newsletter included 50% of RD in cost of goods sold, and Dan (46:33) Mm-hmm. Shimin (46:33) the revenue number is still at 55%. I asked Claude, the irony is not lost on me to include a hundred percent And it is still a fairly healthy 26%. So then I asked it to also subtract the hyperscaler distribution fee, which of course the data centers need to take a cut on on every single token spent as well. And even after all that, it's still nine percent. So there's still like cash flow, they're still making money on on every single token Dan (47:01) Not an imprint, yeah. Shimin (47:03) sort of. Yeah. Even discounting all the new model, yeah. Dan (47:06) model training and yeah. Shimin (47:08) Which looks pretty good. Now it's a lot less rosy than the eighty-five percent, but because I don't think they can ever stop doing Dan (47:19) Mm-hmm. Shimin (47:20) R and D. Right. So I firmly believe that R and D should be included as part of their cost of goods salt. and and that is with a healthy assumption that they can that the R and D amount would only stay relatively steady and not grow. Right. Dan (47:38) Yeah. Well it isn't for tax reasons too, right? 'Cause you can write off the R and D costs. Shimin (47:44) Right. And then the next big thing is they're still expected to lose eighty billion dollars this year because they still have so much compute Dan (47:50) Yeah. Shimin (47:53) outstanding. not great. I believe they still have something like five hundred billion dollars of compute and debt commitments over the next couple of years. So I think they they will need to essentially keep on expanding at the current rate and not plateau for for all of that to be okay. Dan (48:13) Yeah, they need to do to at least one other industry what's happened to tech in order for that to work, is my my takeaway on that. Shimin (48:20) Maybe that's why they recently bought a biolab and paid a one person drug discovery firm. Dan (48:24) It could be, yeah. You know, 'cause it's like, hey, at least there there's it's the closest thing to like a deterministic outcome, right? Like it's either it worked or it didn't in terms of life. Like genetic code that Shimin (48:37) Yeah. Dan (48:39) reproduces or doesn't. Shimin (48:41) I think their total b addressable market, which if you recall from our SpaceX episode was something like twenty trillion. I believe anthropics is listed to be forty trillion, a number that rivals the GDP of this country, which which is not small to begin with. Dan (48:57) Yeah. Shimin (48:58) Yeah, which is pretty wild. and I think the the next eye popping number is their nine hundred percent. net revenue retention. So every single one of their customers are on average paying nine nine X the amount that they were paying last year for for their services. Just the growth end, yes, if if this is not the plateau and this is the beginning of the exponential, then then maybe this company is worth every penny of their one and a half trillion, two trillion expected market cap. Dan (49:29) Well, our our friend Nathan in his blog post about the bimodal outcome of this, right? It's either gonna be really good or really bad. Shimin (49:41) Assuming you're assuming you're a shareholder of of Anthropic and Dan (49:43) Yeah. Which interestingly too, I didn't realize what a big shareholder Amazon is. Like actually quite Shimin (49:51) Mm. Dan (49:52) a few like fangy kind of companies are gonna be making a bunch of money off of this, which is interesting. Shimin (49:58) Yeah. They they all invest in each other, kinda like how Dan (50:00) Mm-hmm. Shimin (50:01) Google had a huge pop with the SpaceX IPO. Any other numbers jumped out at you from the Anthropicus S-1 Dan (50:10) Not particularly. I think at a certain point my brain just stops working at numbers that large, so Shimin (50:16) Yeah, what this number. Dan (50:18) Yeah. Shimin (50:18) what is money? the last thing I want to mention is that just like Facebook, and that's part of the reason why I think Facebook was a good tying, the founders of Anthropic has a long-term benefits trust that holds a single share worth 50.1% of the voting power of the entire company. So essentially, even though Anthropic is already a public benefits corporation, the shareholder still does not have the same kind of voting power that you would expect from a traditional shareholder first company. Like ultimately Dan (50:53) It is. Shimin (50:54) the shareholders will not have much power in on a day-to-day running of the company. So a lot of this is still really unusual despite the precedents that meta I showed us. So now that we finally have our the prince that was promised, the long awaited S1 from one of the Frontier Labs, how do we feel about the clock? Dan (51:16) We did say well we did say November, right, was gonna be when things would be get interesting. So like I think we're does the S one say when it's actually going, when it's happening? I miss that in Google's doing things. Thinking a little longer. Shimin (51:28) Yeah. According to this blog post, they're expected to go public in about three weeks. Dan (51:35) Yeah. Well. Shimin (51:36) it's the mid November surprise. This is the real November October surprise here. It's Dan (51:39) huh. Shimin (51:40) the Anthropic S one. Dan (51:42) Mm. Yeah, it went on Monday, October twenty sixth, November ninth, twenty twenty sixth. Is the road show. Their ticker will be Ant. Shimin (51:50) my take on this is of course we have no idea how the stock will do. And of course, this Dan (51:56) Right. Shimin (51:56) is not financial advice. Listeners do not trade based on this information or our hot takes. but my hot take is because they're expected to go public pretty soon, the the case of the IPO going bad has dramatically. increased in the short term. Dan (52:13) Wait, what? Shimin (52:15) So assume okay, so let's say let's say Anthropic goes IPO in three weeks, and lots of retail investors are not buying the stock 'cause they do not believe in the ability of Anthropic to continue their trajectory of revenue growth. Dan (52:29) And pay this back. Yeah, yeah. Shimin (52:30) Then Anthropic will face a liquidity issue. and it might bring down additional sell-offs. Like I could easily see a case where Anthropic dips 20% that day, and all the other major companies also start dipping Dan (52:44) Mm-hmm. Shimin (52:44) just as a part of the correlation. And then margin calls happen and liquidity crunches here, and we find out what the end game is of this whole thing. Now I'm not saying that's likely. I'm just saying that scenario has moved closer to us. Dan (52:56) Yeah, I was actually true. I was actually thinking it's the other way around, probably, because all the like hype about it. But it really could go either way, you're right. And so yeah, I guess that by that definition we would be inching closer to maybe not inching. We're jumping quickly towards knowing whether or not the bubble's bursting. okay, I buy that. How how far do you want to move it? Shimin (53:20) I'm thinking thirty seconds. Dan (53:22) Two minutes. Two minutes. Shimin (53:24) Not not not a whole two minutes, but like I'm thinking thirty seconds. Yeah. Dan (53:26) The eponymous two minutes to make that Okay, fine. It's a pretty big jump for reason recent moves, so Shimin (53:34) Yeah, that said, this S1 is not it's not a bloodbath, right? Dan (53:38) Yeah. Shimin (53:39) but they're still losing eighty billion dollars a year. Like if they were actually making money, I would be like, let's just throw the clock out. We are not in an A by bubble, like everything is fine. Let's cut the segment, let's just cover the good things from now on. But like, yeah. Dan (53:46) And we're done. Yeah. It's been fun. It's only been a year, but here we are. Shimin (53:54) Like how long can they how long can they keep on losing this much money for Dan (53:58) Mm-hmm. Shimin (53:59) and wielding investors? Buy that the rosy growth picture. Yeah. Dan (54:01) And tolerate it. Yeah, exactly. Cause when you think about it, that was the the situation normal before zirp that's how every company was, at least in tech. Shimin (54:09) Right. Yeah. And we're no longer in Zerpland, so Dan (54:13) It's will they get away with it? Shimin (54:14) Yeah. All right. Moving to three minutes and thirty seconds. And and speaking of of speaking of it's been a year. next week is our one year anniversary of the show. So we have Dan (54:26) Wow. Shimin (54:26) a we're gonna have like a a special looking back episode with special segments looking back at how far things have changed in the last year since we started this show. Dan (54:35) That's funny. Yeah, you should have Claude pull like a whole bunch of predictions that we made that were just like both right and wrong. That'd be kinda interesting. Shimin (54:43) would be great. Yeah. Looking forward to that. okay then. Well, we've set the clock. and that comes that means we've come to the end of the show. So thank you, listeners, for joining us for our study session this week. If you like the show, if you learned something new, please share the show with a friend. You can also leave us a review on Apple Podcasts or Spotify. It helps people to describe the show and we really appreciate it. If you have a segment idea, a question for us or a topic you want us to cover, shoot us an email at humans at adipod.ai. We'd love to hear from you. You can find the full show notes, transcripts, and everything else mentioned today at www.adipod.ai. Thank you again for listening. I will catch you next week. Bye. Dan (55:18) Don't forget to try D Slop and then use it to write Shmin emails. See if he can tell. Shimin (55:22) yes, please do. Okay. Bye.