Latest / AI Fire Daily / #584 Neil: Grok 4.6 Takes On Real Projects and Long AI Workflows

Transcript

#584 Neil: Grok 4.6 Takes On Real Projects and Long AI Workflows · AI Fire Daily

• 0:00
  Imagine stepping away from your computer, beat,
• 0:03
  and while you're gone, an AI is quietly building
• 0:06
  an entire playable 3D physics game from scratch.
• 0:10
  That is just wild to think about. OK, let's unpack
• 0:13
  this. Welcome to the deep dive. We are looking
• 0:15
  at a major shift today. Yeah, specifically Elon
• 0:18
  Musk's XAI and their launch of Grok 4 .6. They
• 0:21
  tend to launch models like SpaceX launches rockets.
• 0:24
  Yeah. Fast, loud, and impossible to ignore. Oh,
• 0:27
  absolutely. But we aren't just looking at a simple
• 0:29
  version update today. Right. We're looking at
• 0:32
  a completely different approach to artificial
• 0:34
  intelligence. What's fascinating here is the
• 0:36
  core evolution of the model itself. Grok is fundamentally
• 0:40
  shifting its purpose. It is moving away from
• 0:43
  just being a standard chat bot, answering a quick
• 0:46
  question or writing a simple email. It is transitioning
• 0:49
  into a system for continuous long -running work.
• 0:51
  Exactly. From a conversational tool to a dedicated
• 0:54
  worker. So to really understand what that means
• 0:57
  for you, we need to look under the hood. We need
• 0:59
  to know why Grok 4 .6 can handle these massive
• 1:02
  tasks. Right, because they definitely change
• 1:04
  the training process quite a bit. They did. This
• 1:06
  is not just a minor version bump. They completely
• 1:09
  rewrote the recipe. They absolutely did. For
• 1:12
  starters, XAI trained Grok 4 .6 much longer than
• 1:16
  4 .5. Wow. But it is not just about the time.
• 1:20
  It is about what they actually fed it. They pumped
• 1:22
  in massive amounts of model -generated reasoning
• 1:25
  data, right? Yeah. They had previous AI models
• 1:28
  generate complex logical puzzles. Then they fed
• 1:31
  that synthetic logic back into Grok. That makes
• 1:33
  the reasoning pathways much more robust. Exactly.
• 1:36
  They also gave it highly advanced technical material
• 1:39
  and top -tier engineering data. And they swapped
• 1:42
  out the optimizer entirely. Ugh. Think of the
• 1:45
  optimizer as the core engine. It updates the
• 1:47
  AI's knowledge during training. Changing that
• 1:49
  fundamental engine builds a much stronger foundational
• 1:52
  base. And they did all of this before fine -tuning
• 1:55
  even started. Right. And the most critical part
• 1:57
  of that foundation is the training style. They
• 2:00
  focused heavily on agentic reinforcement learning.
• 2:02
  Meaning teaching AI by rewarding it for completing
• 2:05
  multi -step goals. Spot on. They trained it by
• 2:07
  giving it increasingly complex tasks. Tasks that
• 2:10
  required serious, long -term planning. Like hardcore
• 2:14
  multi -file coding projects? Exactly. Comprehensive
• 2:17
  web development, real knowledge work, even computer
• 2:20
  -aided design files. The goal was forcing the
• 2:23
  model to handle multi -step work seamlessly.
• 2:25
  It can't just answer one prompt and stop. No,
• 2:28
  it needs to carry a thought process through 50
• 2:30
  different steps. And it has to do that without
• 2:32
  losing its mind. These training changes sound
• 2:35
  highly technical, but the benchmarks make the
• 2:38
  tangible difference much easier to see. Yeah,
• 2:40
  let's look closely at those numbers. They are
• 2:42
  quite revealing for this release. Let's start
• 2:44
  with the Artificial Analysis Intelligence Index.
• 2:46
  Right, a very well regarded aggregate score.
• 2:49
  Grok 4 .6, high scores of 61. That is a solid
• 2:53
  jump from Grok 4 .5 score of 56. It really is.
• 2:56
  And it matches GPT 5 .6 SolMax exactly at 61.
• 3:00
  It sits just one point below Fable 5 Max at 62.
• 3:04
  So it is right there at the absolute bleeding
• 3:06
  edge. But it gets even better from there. Look
• 3:08
  at the GDPVal AA version 2 benchmark. OK. Grok
• 3:12
  4 .6 completely dominates this specific test.
• 3:14
  It reaches a massive 1 ,753 ELO score. For those
• 3:18
  unaware, an ELO score is basically like a chess
• 3:20
  ranking. Yeah, you pit the models directly against
• 3:23
  each other in real world scenarios. And a 1753
• 3:26
  ELO beats Fable 5 Max at 1741. It also easily
• 3:32
  beats GPT 5 .6 Solmax, which sits at 1728. This
• 3:37
  specific result really matters for users. GDPVAL
• 3:41
  focuses heavily on real knowledge work. Exactly.
• 3:43
  It doesn't just test random history trivia. It
• 3:46
  measures actual utility in messy business scenarios.
• 3:49
  But we do have to balance that by looking at
• 3:51
  DeepSWE version 1 .1. Yeah, that benchmark is
• 3:54
  designed purely for coding. It drops the AI into
• 3:57
  a software environment to solve complex issues.
• 3:59
  Croc 4 .6 scores... is 65 .9 % here. That is
• 4:03
  certainly a major upgrade from the previous 54%.
• 4:05
  But it still trails behind the true coding frontier.
• 4:09
  GPT 5 .6 hits a very strong 73%. And Fable 5
• 4:12
  Max sitting at 70%. So Grok 4 .6 definitely has
• 4:15
  room to improve on pure software engineering.
• 4:17
  Well, for sure. But I always think of benchmarks
• 4:19
  like running on a treadmill. The conditions are
• 4:21
  perfectly controlled. There are no surprises.
• 4:22
  But real software engineering is like running
• 4:25
  through a dense forest. Benchmarks miss how well
• 4:29
  an AI keeps context when things get messy. They
• 4:32
  miss the real -world feedback loops. Yeah, absolutely.
• 4:35
  Does obsessing over benchmark scores actually
• 4:37
  distract us from how useful a model truly is?
• 4:41
  I really think so. Benchmarks are just static
• 4:43
  tests measuring isolated logic points. True usefulness
• 4:47
  comes from adapting dynamically over time. If
• 4:50
  an AI writes code, sees an error, and figures
• 4:53
  it out based on something from 20 minutes ago,
• 4:55
  a benchmark just doesn't capture that. Right.
• 4:57
  Real projects test context retention, not just
• 5:00
  static trivia. Exactly. So we have to look at
• 5:02
  the ultimate software stress test. Throwing a
• 5:04
  massive blank canvas project at the model. Yes.
• 5:07
  We have two incredible real world tests to look
• 5:10
  at here. Test one is building a video game directly
• 5:13
  inside Grok Build. Which is their brand new integrated
• 5:16
  development environment. The prompt asks for
• 5:18
  a first person 3D puzzle game. Heavy inspiration
• 5:20
  from the classic game Portal. And that means
• 5:23
  handling incredibly complex physical mechanics.
• 5:25
  It has to figure out how to place connected portals
• 5:28
  on valid walls. And more importantly, it has
• 5:31
  to preserve a player's exact momentum. Right.
• 5:34
  Imagine falling deeply into a portal on the floor.
• 5:38
  The AI must calculate that speed. then launch
• 5:41
  you out of a wall portal at that exact same velocity.
• 5:44
  For an AI predicting the next word, translating
• 5:47
  that into spatial physics is mind -bending. It
• 5:50
  also must maintain correct camera direction smoothly.
• 5:54
  It handles 3D collisions without clipping through
• 5:56
  walls. It needs to calculate dynamic lighting
• 5:58
  and cast shadows correctly. And write the math
• 6:02
  for all of this from scratch. The rule for this
• 6:04
  test is incredibly strict, too. It must fix broken
• 6:07
  mechanics before adding any visual polish. The
• 6:10
  gravity and momentum must work flawlessly before
• 6:12
  making the walls look pretty. Yeah. Now, test
• 6:15
  two is a bit different. But arguably more relevant
• 6:19
  to the average listener. It is building a complete
• 6:21
  SaaS website. A project management app called
• 6:24
  FlowTask. This is a highly practical, everyday
• 6:27
  test. It needs a polished landing page with a
• 6:29
  hero section. A simple pricing tier section and
• 6:32
  a comprehensive FAQ. It also needs a fully functional
• 6:35
  app dashboard. We are talking a sidebar menu,
• 6:39
  project overviews, and simple analytics. And
• 6:41
  the kicker is that it must actually work in practice.
• 6:44
  Right. It's not just a pretty picture. It needs
• 6:47
  realistic sample data populating the dashboard.
• 6:50
  The forms, search bars, and data filters must
• 6:53
  function properly. If you click a filter button,
• 6:55
  the screen actually has to update. And it has
• 6:58
  to be perfectly responsive across all devices.
• 7:00
  Yeah. Desktop, tablet, And mobile. I have to
• 7:03
  say, I still wrestle with prompt drift myself.
• 7:06
  Beat. I'll ask an AI to write a Python script.
• 7:10
  By step five, it forgets what database we're
• 7:12
  using and hallucinates features. Oh, it happens
• 7:15
  to everyone. I once asked an AI to build a landing
• 7:18
  page. By the third prompt, it decided I needed
• 7:20
  a complex cryptocurrency checkout system. Exactly.
• 7:23
  You ask for too many steps and the AI loses the
• 7:26
  plot. It just loses the thread entirely. That's
• 7:28
  why Grok maintaining intense focus here is really
• 7:31
  impressive. It holds the entire project architecture
• 7:33
  in its mind. Remembering the original goal while
• 7:36
  fixing tiny bugs along the way? But let me ask
• 7:38
  you this. Why is simulating physics in a game
• 7:41
  a better test of an AI's brain than asking it
• 7:45
  to write an essay? Because physics demands continuous
• 7:49
  unbroken spatial reasoning. If an AI writes a
• 7:52
  weak sentence in an essay, you just keep reading.
• 7:54
  It still functions. But if gravity fails for
• 7:57
  a single frame in a game or a collision mesh
• 7:59
  has a tiny math error, the entire game crashes
• 8:03
  instantly. There is zero margin for error. Exactly.
• 8:06
  Physics requires precise logic over time, not
• 8:08
  just text generation. Yeah. Mineral sponsor red.
• 8:11
  provided separately. So we have seen how it handles
• 8:14
  standard controlled prompts. But what happens
• 8:16
  when the wider development community gets their
• 8:18
  hands on it? People are pushing it incredibly
• 8:20
  hard and the results are fascinating. How does
• 8:23
  it survive wild, unpredictable code bases? Well,
• 8:27
  Chang you testing Grok with a cozy 3D Unity game,
• 8:30
  the focus was strictly on the camera system.
• 8:32
  Watching how real -time interactions flow as
• 8:34
  the player moves around. Yeah, and that real
• 8:36
  -time flow is absolutely critical. You notice
• 8:38
  small computational problems quickly when things
• 8:41
  move dynamically. A static screenshot of code
• 8:44
  hides subtle structural flaws. Right, and then
• 8:47
  Daniel Frena took it much further. He made Grok
• 8:50
  build a fast -paced snowboarding game from scratch.
• 8:52
  That is a brilliant and punishing test. Fast
• 8:55
  movement severely stresses any physics engine.
• 8:57
  Calculating terrain collisions in milliseconds.
• 9:01
  If physics break at high speeds, visuals won't
• 9:03
  save it. But out of all the community tests...
• 9:06
  The bug hunt benchmark is the craziest. Oh, we
• 9:09
  heard and tested Grok 4 .6 on a massive scale.
• 9:12
  A brutal benchmark featuring 105 deeply hidden
• 9:16
  bugs. And these are spread across two massive
• 9:18
  real -world repositories. Literally like finding
• 9:21
  a needle in an interconnected haystack. You have
• 9:23
  to understand how file A affects file Z. Exactly.
• 9:27
  And at the highest reasoning level, Grok 4 .6
• 9:29
  succeeded wildly. Out of the 105 hidden bugs
• 9:32
  in the key, it found 27. For context, Grok 4
• 9:35
  .5 could only find 17 on this test. That is impressive.
• 9:39
  But this is where the story gets truly strange.
• 9:41
  Yeah, it really does. While searching for those
• 9:43
  27 bugs, it found 15 completely real hidden bugs.
• 9:47
  Bugs the human test creators hadn't even included
• 9:49
  in the answer key. Let's just pause on that.
• 9:53
  Beat. Plus. It fixed one specific bug that 13
• 9:57
  other models had completely failed to solve.
• 9:59
  Across 19 earlier runs, the AI didn't just take
• 10:03
  the test. It audited the test itself. It found
• 10:05
  flaws that senior developers missed. This is
• 10:07
  a profound shift in capability. It isn't just
• 10:10
  grading against a rubric. It's finding the flaws
• 10:12
  in the rubric itself, like a senior engineer
• 10:15
  reviewing code. Rather than a student guessing
• 10:18
  the right answer. But how does an AI manage to
• 10:20
  find a bug that isn't even documented in the
• 10:22
  test's answer key? It is no longer just skimming
• 10:25
  for typos. It looks deeply at the execution flow.
• 10:28
  It traces the actual data path. It sees where
• 10:31
  the mathematical outcome contradicts the structural
• 10:33
  logic. It realizes a function will return a null
• 10:36
  value based on a database structured three files
• 10:38
  over. It understands the holistic system. Yeah,
• 10:41
  it actually understands the code's intent rather
• 10:43
  than just pattern matching. Precisely. So we
• 10:46
  know it can build digital worlds. We know it
• 10:47
  can hunt deep bugs. The practical question for
• 10:50
  you, the listener, is clear. What is the barrier
• 10:53
  to entry here? The pricing structure is remarkably
• 10:56
  simple. It is based on tokens. Chunks of words
• 10:59
  or pieces of code. Right. It costs $2 per 1 million
• 11:04
  input tokens. The text or code you feed into
• 11:06
  it. And it costs $6 per 1 million output tokens.
• 11:10
  The code it generates back for you. Output is
• 11:12
  always more expensive. Generating new logic requires
• 11:15
  vastly more computing power. And there is a fast
• 11:18
  variant available, right? Yes, for when you need
• 11:21
  answers instantly. That costs double the standard
• 11:24
  price. Where can people actually use it? It is
• 11:27
  fully integrated into Cursor right now. A very
• 11:29
  popular AI code editor. It is on Grok Build.
• 11:32
  You can use the XAI API directly. It's also on
• 11:36
  OpenRooter, Vercell, and Cloudflare. Very wide
• 11:38
  availability right out of the gate. But let's
• 11:40
  be honest about the target demographic. Who is
• 11:42
  this actually built for? Let's break down the
• 11:44
  specific user profiles. It is highly valuable
• 11:47
  for dedicated developers. They need this context
• 11:49
  retention for massive code bases. It is absolutely
• 11:52
  perfect for builders, people prototyping games,
• 11:55
  or SaaS platforms. And it is explicitly built
• 11:57
  for agent users. People running long, complicated,
• 12:02
  multi -step workflows, workflows that might take
• 12:05
  hours. But it is not necessary for the casual
• 12:09
  user. Definitively not. If you just want to brainstorm
• 12:12
  ideas or summarize an article, you do not need
• 12:15
  this. Even the fast variant is extreme overkill
• 12:18
  for daily tasks. Oh yeah, completely unnecessary.
• 12:22
  Tools are only as good as the problems you have
• 12:24
  to solve. Using Grok 4 .6 to write a grocery
• 12:27
  list is like driving a Formula One car to the
• 12:30
  supermarket. That is a great way to put it. Sure,
• 12:33
  it's an incredible machine. But it's built for
• 12:35
  high -speed performance on a track. If you drive
• 12:37
  it to the grocery store, the engine literally
• 12:39
  overheats. Because it's going too slow. And there's
• 12:41
  no trunk for your groceries anyway. You're wasting
• 12:43
  massive compute on simple text generation. Should
• 12:46
  a casual user trying to learn a new topic even
• 12:49
  bother opening the paid interface? Probably not.
• 12:52
  The free tools handle basic learning and conversational
• 12:55
  brainstorming perfectly fine. You only need these
• 12:58
  high -tier tools for complex, persistent problem
• 13:00
  solving, requiring hundreds of pages of context.
• 13:04
  Basically, stick to free tools unless you're
• 13:06
  building multi -step projects. Exactly. So as
• 13:09
  we wrap this up, we need to connect these specific
• 13:11
  use cases to a much grander vision. We have to
• 13:14
  look at where XAI is heading next. The big idea
• 13:17
  here isn't just the Grok 4 .6 model itself. No,
• 13:20
  the real story is the ecosystem. Grokbot and
• 13:24
  Grok AI agents. This is a massive shift in the
• 13:27
  paradigm of computing. We are moving rapidly
• 13:29
  toward AI that holds incredibly deep context.
• 13:32
  It can take a highly complex task. and just keep
• 13:34
  working in the cloud. It doesn't start from zero
• 13:37
  every single time you open a chat window. Right.
• 13:39
  And it changes the workflow completely. You hand
• 13:41
  over a massive, tedious task and you go to sleep.
• 13:45
  Like refactoring an entire database. The agent
• 13:47
  just keeps building, testing, failing, fixing,
• 13:50
  and iterating. While you are completely disconnected.
• 13:52
  Yeah. Whoa. Two -sex silence. Imagine an agent
• 13:57
  that just keeps coding in the cloud, expanding
• 13:59
  a software universe long after you've closed
• 14:01
  your laptop. It is exactly like having a tireless,
• 14:04
  brilliant digital colleague. It never stops iterating.
• 14:07
  It never loses the thread because it got distracted.
• 14:10
  It just keeps pushing the project forward. Which
• 14:12
  is incredible. Yeah. But it also forces us to
• 14:14
  ask a difficult question as we look to the future.
• 14:17
  If an AI agent becomes reliable enough to do
• 14:21
  the heavy lifting of software engineering while
• 14:23
  you sleep, beat, do your role shift fundamentally?
• 14:28
  That is the big question. Do you transition from
• 14:30
  being a builder who writes every line to simply
• 14:33
  being an editor of grand ideas? Wow. And ultimately,
• 14:36
  are we truly ready to trust the foundation of
• 14:39
  our digital world to code that we didn't actually
• 14:41
  write? That is something we all need to think
• 14:43
  about. Thank you for taking this deep dive with
• 14:45
  us today.