Latest / Lenny’s AI Builders / i gave GPT-6 Astra and Fable 5.1 the same video to edit
Transcript
- I gave ChatGPT 6 Astra and Claude Fable 5.1 the same video to edit. We are not basing this off vibes. We are doing this shit properly. We have 13 videos. Astra created three short form and three long form. Fable created three short form and three long form. And then I've created one long form. I'm going to go and score each video out of 100. Then at the end, we'll add those scores up and we'll see who is the best video editor. Astra, ChatGPT 6 Astra, Fable 5.1, me. I gave each model the same two prompts. This is the first prompt I gave Astra. If you're interested in that, pause the video now to take a look. And the first prompt that I gave Fable was basically the same one. All up, that first version took Astra about an hour and a half, but I had to nudge it several times. But what about Fable 5.1 for version one of this video? It worked for three hours, 34 minutes and 28 seconds. And I didn't have to nudge it once. It just went for it. And it also created and looped eight different times. At this point, I just preferred how Fable 5.1 did this because I just want to give the prompt to the AI and then come back and it'd be done. I don't want it to stop several times to ask me for permission to do something stupid. That really annoyed me about Astra, but I didn't have that problem at all with Fable 5.1. But what about the second version? Once again, I gave Astra and Fable 5.1 the same prompt, and it basically says, make me something beautiful and breathtaking. Spend most of your time on the first 30 seconds to a minute of the video, include lots of really nice motion graphics, and basically just show me what you're capable of as an AI model. Don't hold back. Importantly though, I also explicitly told each model to loop to create a 100 point rubric and to loop until it achieves that score of 100 out of 100. And Astra completed version two of the video in 30 minutes and 34 seconds. Kind of. It finished at an independent score of 80 out of 100, even though I told it to loop until 100 out of 100. And instead of just continuing to execute the goal, which is what it should do, as we can see, it just stalled. So Astra spent about 30 minutes on version two and then just gave up. What about Fable for version two? Same prompt that I gave Astra, but check out how long it took. Six hours, 26 minutes, and 48 seconds. Fable 5.1 did eight render loops, and it says the scores went 72 out of 100, and then 87, 95, and 100. I didn't have to nudge Fable 5.1 at all. So why the fuck does any of that matter? Fable 5.1 is an AI model that you just give a task to, a big task, and expect it to just go away and work for hours on end. So here is our first video and this is a short form. This is what it has created for us. But the point that this is getting across is that... This thing can control a computer and a browser better than you can now. It's pretty mid. Personal taste, five out of 10. Motion and visual craft, two out of 10. Opening and payoff, two. Story and retention, three. Transitions, one. Sound, four. Captions and readability. I'd like the captions to be a bit bigger. Let's go with five. Here is the second short. What the hell? Whoa. Number five, make an original soundtrack by painting music onto a grid. I'll build a melody, add bass and drums, then play it. That one kind of had me for some reason. So for story and retention, I'm going to go for like an eight. This is clearly a different model to the one we just saw because the styling of it is different. Sound is good. It's actually got some music in there. So I'm going to go for six. Captions are not bad, but I'd still like them a bit bigger. Six. And we've got 58 out of 100 for that one. Let's go to short form number three. What do I mean by that? I had never heard of this application before. And if you had shown me this application an hour ago, I would have had no idea how to use it. Kind of mid. There's a weird three or four second part at the end where I'm not doing anything, and my screen just stays exactly the same, same at the bottom. Personal taste, five. Sound, four. Captions and readability, four. For a total score of 31 out of 100. Our fourth short form video. And this is the version that Astra just created. This one prompt got 1,255 followers in seven days. If you've been creating... So now I want to see whether Astra can replace my girlfriend. All right, this one was better with showing stuff that was on my screen. And it's also gone ahead and blurred some stuff out. I don't know why, but I think that deserves a few points. So that's 54.5 out of 100. Now onto the fifth video. Then here I am vibe coding at 2.45 AM on a Saturday morning. And I look at the model picker, hoping to see Astra there. I was feeling lucky. And then boom, here it is. what we're going to have a look at is the thing that everyone has been talking about when it comes to how good Astra is, which is this thing, computer use. So not bad. The motion graphics were actually really good. The editing was a little bit off though. I felt like it included a few too many word gaps. Captions and readability. I'm gonna go for seven because I liked how it actually zoomed in on things. That one has 59 out of 100. And here is the next long form video. And this one is 11 minutes and 39 seconds long. What do I mean by that? I had never heard of this application before. And if you had shown me this application an hour ago, I... would have had no idea how to use it. But specifically, we're going to be taking a look at the top five use cases of Astra. Motion graphics are again really good, but the story to me just didn't really make any sense and it felt really disjointed. Personal taste, five. Motions, motion and visual craft, eight. Opening and payoff, five. Story and retention, three. Captions and readability, five. Final score 50.5 out of 100. The next long form is 19 minutes. This thing can control a computer and a browser better than you can now. And it's not even close. But I hate to break it to you. your taste still matters. Story makes more sense and the motion graphics are also pretty good too. I think that might be my favorite one so far. So that I believe is our highest score so far at 65 out of 100. Next long form is 11 minutes and 37 seconds long. So now I want to see whether Astra can replace my girlfriend. My girlfriend, shout out Yassy, she edits my YouTube videos for me. Tools like Descript make video editing so much easier than what it used to be, but it still takes hours. All right, I'm noticing with this model, whichever one it is, that the motion graphics is really good and it's got music, but the story that it's edited together just seems really disjointed. That's at a score of 50 out of 100. Next long form is 18 minutes and 59 seconds long. One use case above all the others that has not yet been shown. What is that use case? Video editing. So now I want to see whether Astra can replace my girlfriend. Similar to the one that we watched before that got 59 out of 100. The next one is eight minutes and 59 seconds long. Whoa. ChatGPT 6 Astra just dropped. We're not going to look at the benchmarks. You already know this is a fucking amazing model. What we're going to have a look at is the thing that everyone has been talking about when it comes to how good Astra is, which is computer use. That's the one that I edited, and I actually uploaded that one to YouTube just a few hours ago. Take these ratings with a pinch of salt because it is me scoring me. Motion and visual craft. See, it doesn't have the same motion graphics as the other two, but personally, I think the visual craft is much better than the other two. Let's go for seven. Captions and readability. 74.5 out of 100. We have three more videos to take a look at. Third last one is a short. I'm building this in layers so the changes… Whoa. What the hell? Whoa. Okay, number five. Make an original soundtrack by painting music onto a grid. I'll build a melody, add bass and drums, then play it. That one was not too bad because it shows some cool moments and it also sped up and showed a little caption that read 8X. And I thought that was a nice touch. Personal taste, seven. Total score, 61.5 out of 100. Second to last video, short form, 24 seconds. Yes, Astra is amazing at computer use. Astra could do the bare minimum there. It could remove the word gaps and it could remove the retakes. It could apply the captions. and it could resize my head. Yeah, I didn't like that one. I didn't know what it was saying. And the editing was like the layout, pretty shit. Not a fan. Total score 28 out of 100. And lastly, our final video, which is a long form and is 11 minutes and 35 seconds. Well, it's just duplicated the slide and holy shit. I don't need to know how to use that because of Astra. What the hell? Whoa. But specifically, we're going to be taking a look at the top five use cases of Astra when it comes to computer use. The story doesn't really make sense to me, but the motion graphics are cool. So for personal taste, four out of 10, motion and visual craft, six, opening and payoff. I thought that was actually pretty good. It showed me some highlights of the whole video. We're going to go seven there. 47.5 out of 100. So that is all the videos. For the final verdict, I got to stand up. So the average score for the long form videos was 57.9 out of 100. The average score for the short form videos is 44.2 out of 100. And the total average score is 51 out of 100. But the question still remains, which one was Fable 5.1 and which one was GPT-6 Astra? Let's take a look. The verdict. Okay. Fable 5.1, long form. Damn. It got 61 out of 100. And for short form, it got 49.3 out of 100. for a total score of 55.2. GPT-6 Astra, 49.3 out of 100 for long form, 39 out of 100 for short form, with an average of 44.2. So all in all, where does that leave us? I'd say the verdict is Human Editor still comes first, but in all honesty, I would give it less than 12 months before that changes. After the Human Editor comes Fable 5.1, and in third place is GPT-6 Astra. Comment below which use case you want me to compare Fable 5.1 and GPT-6 Astra against next. And if you liked this video, hit that subscribe button so you can get more just like it. And I'll see you soon.