Latest / AGI Dreams – Open, Uncensored, & Local - AI News Digest / AGI Dreams Podcast – May 13, 2026

Transcript

AGI Dreams Podcast – May 13, 2026 · AGI Dreams – Open, Uncensored, & Local - AI News Digest

• 0:00
  The
• 0:00
  supply
• 0:00
  chain
• 0:01
  worm
• 0:01
  known
• 0:01
  as
• 0:01
  Shai-Hulud
• 0:02
  —
• 0:03
  named,
• 0:03
  with
• 0:04
  apparent
• 0:04
  literary
• 0:04
  flair,
• 0:05
  after
• 0:05
  the
• 0:05
  sandworms
• 0:06
  of
• 0:06
  Dune
• 0:06
  —
• 0:07
  has
• 0:07
  burrowed
• 0:07
  deeper.
• 0:09
  Socket
• 0:09
  Security
• 0:10
  confirmed
• 0:10
  that
• 0:10
  newly
• 0:10
  compromised
• 0:11
  packages
• 0:11
  now
• 0:12
  include
• 0:12
  @opensearch-project/opensearch
• 0:14
  (versions
• 0:15
  3.5.3
• 0:16
  through
• 0:16
  3.8.0,
• 0:18
  pulling
• 0:19
  1.3
• 0:19
  million
• 0:20
  weekly
• 0:20
  downloads),
• 0:21
  mistralai
• 0:21
  2.4.6
• 0:23
  on
• 0:23
  PyPI,
• 0:23
  and
• 0:24
  guardrails-ai
• 0:24
  0.10.1
• 0:26
  on
• 0:26
  PyPI.
• 0:27
  The
• 0:27
  guardrails-ai
• 0:28
  package
• 0:29
  executes
• 0:29
  malicious
• 0:29
  code
• 0:30
  on
• 0:30
  import,
• 0:31
  downloading
• 0:31
  a
• 0:31
  payload
• 0:32
  from
• 0:32
  git-tanstack[.]com
• 0:33
  and
• 0:33
  running
• 0:33
  it
• 0:33
  without
• 0:34
  integrity
• 0:34
  verification.
• 0:35
  The
• 0:35
  attackers
• 0:36
  left
• 0:36
  a
• 0:36
  calling
• 0:36
  card:
• 0:37
  the
• 0:37
  domain
• 0:38
  displayed
• 0:38
  a
• 0:38
  message
• 0:38
  signed
• 0:39
  "With
• 0:39
  Love
• 0:39
  TeamPCP,"
• 0:40
  alongside
• 0:41
  a
• 0:41
  boast
• 0:41
  about
• 0:41
  stealing
• 0:41
  credentials
• 0:42
  for
• 0:42
  over
• 0:42
  two
• 0:42
  hours.
• 0:43
  Meanwhile,
• 0:44
  model
• 0:44
  repositories
• 0:45
  are
• 0:45
  getting
• 0:45
  the
• 0:45
  same
• 0:46
  treatment
• 0:46
  as
• 0:46
  package
• 0:46
  registries.
• 0:47
  A
• 0:47
  fake
• 0:48
  "model"
• 0:48
  called
• 0:49
  Open-OSS/privacy-filter
• 0:51
  appeared
• 0:51
  on
• 0:51
  Hugging
• 0:51
  Face,
• 0:52
  presenting
• 0:53
  itself
• 0:53
  as
• 0:53
  an
• 0:53
  OpenAI
• 0:54
  privacy
• 0:54
  filter.
• 0:55
  It
• 0:55
  is
• 0:55
  actually
• 0:56
  a
• 0:56
  Python-based
• 0:56
  dropper
• 0:57
  that
• 0:57
  downloads
• 0:57
  a
• 0:57
  malicious
• 0:58
  PowerShell
• 0:58
  command,
• 0:59
  which
• 0:59
  spawns
• 1:00
  another
• 1:00
  PowerShell
• 1:00
  instance,
• 1:01
  which
• 1:01
  downloads
• 1:02
  a
• 1:02
  Rust-compiled
• 1:03
  infostealer
• 1:03
  targeting
• 1:04
  Chrome
• 1:04
  credentials,
• 1:05
  WinSCP
• 1:05
  sessions,
• 1:06
  and
• 1:06
  more.
• 1:07
  The
• 1:07
  chain
• 1:08
  is
• 1:08
  loader.py
• 1:09
  to
• 1:09
  base64-encoded
• 1:10
  URL
• 1:11
  to
• 1:11
  PowerShell
• 1:11
  batch
• 1:12
  file
• 1:12
  to
• 1:13
  yet
• 1:13
  another
• 1:13
  base64
• 1:14
  PowerShell
• 1:14
  script
• 1:15
  to
• 1:15
  the
• 1:15
  final
• 1:15
  compiled
• 1:16
  binary
• 1:16
  —
• 1:17
  layers
• 1:17
  of
• 1:17
  obfuscation
• 1:18
  that
• 1:18
  would
• 1:18
  be
• 1:18
  comical
• 1:19
  if
• 1:19
  the
• 1:19
  package
• 1:20
  hadn't
• 1:20
  accumulated
• 1:20
  244,000
• 1:22
  downloads
• 1:22
  before
• 1:22
  being
• 1:23
  reported.
• 1:24
  The
• 1:24
  accounts
• 1:24
  liking
• 1:25
  the
• 1:25
  model
• 1:25
  show
• 1:25
  patterns
• 1:26
  consistent
• 1:26
  with
• 1:26
  coordinated
• 1:27
  bot
• 1:27
  activity,
• 1:28
  some
• 1:28
  already
• 1:28
  closed
• 1:29
  for
• 1:29
  violations
• 1:30
  yet
• 1:30
  still
• 1:30
  boosting
• 1:30
  new
• 1:31
  malware.
• 1:32
  At
• 1:32
  press
• 1:32
  time,
• 1:33
  the
• 1:33
  repo
• 1:33
  remained
• 1:33
  live.
• 1:34
  The
• 1:35
  absurdity
• 1:35
  of
• 1:35
  modern
• 1:36
  supply
• 1:36
  chain
• 1:36
  security
• 1:37
  is
• 1:37
  best
• 1:37
  captured
• 1:37
  by
• 1:38
  a
• 1:38
  satirical
• 1:38
  incident
• 1:38
  report
• 1:39
  making
• 1:39
  the
• 1:39
  rounds.
• 1:40
  "CVE-2024-YIKES"
• 1:45
  chronicles
• 1:45
  a
• 1:45
  fictional
• 1:46
  cascade
• 1:46
  where
• 1:47
  a
• 1:47
  compromised
• 1:47
  npm
• 1:48
  dependency
• 1:48
  leads
• 1:49
  to
• 1:49
  credential
• 1:49
  theft,
• 1:50
  which
• 1:50
  enables
• 1:51
  a
• 1:51
  supply
• 1:51
  chain
• 1:51
  attack
• 1:52
  on
• 1:52
  a
• 1:52
  Rust
• 1:52
  compression
• 1:53
  library,
• 1:53
  which
• 1:54
  gets
• 1:54
  vendored
• 1:54
  into
• 1:54
  a
• 1:55
  Python
• 1:55
  build
• 1:55
  tool,
• 1:56
  which
• 1:56
  ships
• 1:56
  malware
• 1:57
  to
• 1:57
  approximately
• 1:57
  four
• 1:58
  million
• 1:58
  developers
• 1:58
  —
• 1:59
  before
• 2:00
  being
• 2:00
  inadvertently
• 2:00
  patched
• 2:01
  by
• 2:01
  an
• 2:01
  unrelated
• 2:02
  cryptocurrency
• 2:02
  mining
• 2:03
  worm.
• 2:04
  Every
• 2:04
  individual
• 2:05
  failure
• 2:05
  mode
• 2:05
  is
• 2:05
  real:
• 2:06
  hardware
• 2:07
  2FA
• 2:07
  keys
• 2:08
  going
• 2:08
  missing,
• 2:09
  Google
• 2:09
  AI
• 2:09
  Overviews
• 2:10
  linking
• 2:10
  to
• 2:10
  phishing
• 2:10
  sites,
• 2:11
  Dependabot
• 2:12
  auto-merging
• 2:12
  malicious
• 2:13
  PRs
• 2:14
  because
• 2:14
  CI
• 2:14
  passes
• 2:15
  after
• 2:15
  the
• 2:15
  malware
• 2:15
  installs
• 2:16
  the
• 2:16
  test
• 2:16
  runner.
• 2:17
  "Total
• 2:18
  machines
• 2:18
  saved
• 2:18
  by
• 2:18
  a
• 2:18
  cryptocurrency
• 2:19
  worm:
• 2:20
  also
• 2:20
  estimated
• 2:21
  4.2
• 2:21
  million.
• 2:22
  Net
• 2:23
  security
• 2:23
  posture
• 2:24
  change:
• 2:24
  uncomfortable."
• 2:26
  On
• 2:26
  the
• 2:26
  defense
• 2:26
  side,
• 2:27
  Root
• 2:27
  Evidence's
• 2:28
  Robert
• 2:28
  Hansen
• 2:28
  proposes
• 2:29
  a
• 2:29
  provocative
• 2:29
  strategy:
• 2:30
  use
• 2:31
  AI
• 2:31
  to
• 2:31
  rewrite
• 2:32
  third-party
• 2:32
  libraries
• 2:33
  entirely,
• 2:34
  eliminating
• 2:34
  CVEs
• 2:35
  by
• 2:35
  removing
• 2:36
  the
• 2:36
  vulnerable
• 2:36
  code
• 2:37
  from
• 2:37
  your
• 2:37
  codebase.
• 2:38
  The
• 2:38
  tool,
• 2:39
  Lib-Theseus
• 2:40
  (after
• 2:40
  the
• 2:40
  ship
• 2:40
  of
• 2:40
  Theseus
• 2:41
  paradox),
• 2:42
  is
• 2:42
  a
• 2:42
  Claude
• 2:42
  Code
• 2:43
  /
• 2:43
  Codex
• 2:43
  skill
• 2:43
  that
• 2:44
  identifies
• 2:44
  dependencies,
• 2:45
  studies
• 2:46
  their
• 2:46
  actual
• 2:46
  usage,
• 2:47
  generates
• 2:48
  tests,
• 2:49
  writes
• 2:49
  replacement
• 2:49
  code,
• 2:50
  and
• 2:50
  cleans
• 2:50
  up
• 2:51
  artifacts.
• 2:52
  In
• 2:52
  testing,
• 2:53
  it
• 2:53
  rewrote
• 2:53
  14
• 2:54
  of
• 2:54
  18
• 2:54
  modules
• 2:55
  (78%),
• 2:56
  with
• 2:57
  only
• 2:57
  platform-critical
• 2:58
  packages
• 2:58
  like
• 2:58
  Electron
• 2:59
  left
• 2:59
  untouched.
• 3:01
  The
• 3:01
  upsides
• 3:01
  are
• 3:01
  clear:
• 3:02
  no
• 3:02
  vulnerable
• 3:03
  library
• 3:03
  means
• 3:04
  no
• 3:04
  CVE,
• 3:05
  no
• 3:05
  future
• 3:06
  supply
• 3:06
  chain
• 3:06
  compromise,
• 3:07
  and
• 3:07
  potentially
• 3:08
  fewer
• 3:08
  licensing
• 3:09
  headaches.
• 3:10
  The
• 3:10
  downsides
• 3:11
  are
• 3:11
  equally
• 3:11
  real:
• 3:12
  npm
• 3:13
  audit
• 3:13
  cannot
• 3:14
  find
• 3:14
  vulnerabilities
• 3:15
  it
• 3:15
  does
• 3:15
  not
• 3:15
  know
• 3:15
  exist,
• 3:16
  and
• 3:16
  AI-generated
• 3:17
  replacements
• 3:18
  could
• 3:18
  introduce
• 3:18
  novel
• 3:19
  bugs.
• 3:21
  The
• 3:22
  Internet
• 3:22
  Cleanup
• 3:22
  Foundation
• 3:23
  launched
• 3:23
  SecurityBaseline.eu,
• 3:24
  a
• 3:24
  transparency
• 3:25
  platform
• 3:26
  auditing
• 3:26
  the
• 3:26
  cybersecurity
• 3:27
  posture
• 3:27
  of
• 3:27
  European
• 3:27
  government
• 3:28
  websites
• 3:28
  across
• 3:28
  32
• 3:29
  countries,
• 3:30
  67,000
• 3:30
  local
• 3:31
  governments,
• 3:32
  and
• 3:32
  roughly
• 3:32
  200,000
• 3:33
  domains.
• 3:34
  The
• 3:34
  project
• 3:34
  expands
• 3:35
  the
• 3:35
  Dutch
• 3:35
  "Basisbeveiliging"
• 3:36
  initiative,
• 3:37
  which
• 3:37
  has
• 3:37
  been
• 3:37
  measuring
• 3:37
  government
• 3:38
  website
• 3:38
  security
• 3:38
  for
• 3:39
  over
• 3:39
  a
• 3:39
  decade,
• 3:40
  to
• 3:40
  the
• 3:40
  entire
• 3:40
  EU
• 3:41
  and
• 3:41
  EEA.
• 3:42
  Three
• 3:42
  months
• 3:42
  before
• 3:42
  launch,
• 3:43
  the
• 3:43
  foundation
• 3:44
  sent
• 3:44
  tens
• 3:44
  of
• 3:44
  thousands
• 3:44
  of
• 3:45
  advance
• 3:45
  notification
• 3:45
  emails
• 3:46
  to
• 3:46
  affected
• 3:46
  governments.
• 3:47
  The
• 3:47
  results
• 3:47
  suggest
• 3:48
  those
• 3:48
  emails
• 3:48
  were
• 3:48
  largely
• 3:49
  ignored.
• 3:49
  Three
• 3:50
  findings
• 3:50
  stand
• 3:50
  out.
• 3:51
  First,
• 3:51
  3,081
• 3:52
  government
• 3:52
  websites
• 3:53
  place
• 3:53
  tracking
• 3:53
  cookies
• 3:53
  on
• 3:53
  visitors
• 3:54
  without
• 3:54
  consent
• 3:54
  —
• 3:55
  a
• 3:55
  straightforward
• 3:56
  GDPR
• 3:56
  violation.
• 3:57
  YouTube
• 3:57
  embeds
• 3:58
  are
• 3:58
  the
• 3:58
  primary
• 3:59
  offender
• 3:59
  at
• 3:59
  2,077
• 4:00
  tracking
• 4:01
  cookies,
• 4:02
  followed
• 4:02
  by
• 4:02
  Google
• 4:02
  Ads
• 4:03
  at
• 4:03
  842,
• 4:04
  Facebook
• 4:04
  at
• 4:04
  293,
• 4:06
  and
• 4:06
  TikTok
• 4:06
  at
• 4:06
  20.
• 4:07
  Slovakia
• 4:08
  leads
• 4:08
  at
• 4:08
  nearly
• 4:08
  10%
• 4:09
  of
• 4:09
  government
• 4:09
  sites,
• 4:10
  followed
• 4:10
  by
• 4:10
  Greece
• 4:10
  at
• 4:11
  8%
• 4:11
  and
• 4:11
  Portugal
• 4:12
  at
• 4:12
  7.6%.
• 4:13
  The
• 4:13
  foundation
• 4:14
  notes
• 4:14
  these
• 4:14
  are
• 4:14
  mostly
• 4:15
  side
• 4:15
  effects
• 4:15
  of
• 4:15
  embedding
• 4:15
  modern
• 4:16
  web
• 4:16
  components
• 4:16
  without
• 4:17
  understanding
• 4:17
  the
• 4:17
  surveillance
• 4:17
  implications.
• 4:18
  Second,
• 4:19
  over
• 4:19
  1,070
• 4:20
  phpMyAdmin
• 4:21
  database
• 4:21
  administration
• 4:22
  portals
• 4:22
  are
• 4:22
  publicly
• 4:22
  reachable
• 4:23
  across
• 4:23
  EU
• 4:23
  government
• 4:24
  domains.
• 4:24
  France
• 4:25
  leads
• 4:25
  with
• 4:25
  513
• 4:26
  exposed
• 4:26
  instances,
• 4:27
  Poland
• 4:28
  with
• 4:28
  499,
• 4:29
  Hungary
• 4:29
  with
• 4:29
  368.
• 4:31
  Two
• 4:31
  of
• 4:31
  these
• 4:31
  panels
• 4:32
  sit
• 4:32
  at
• 4:32
  CSIRT
• 4:32
  addresses
• 4:32
  —
• 4:33
  the
• 4:33
  organizations
• 4:34
  theoretically
• 4:34
  responsible
• 4:35
  for
• 4:35
  preventing
• 4:35
  exactly
• 4:36
  this
• 4:36
  kind
• 4:36
  of
• 4:36
  exposure.
• 4:37
  The
• 4:37
  foundation
• 4:38
  found
• 4:38
  zero
• 4:38
  financial
• 4:39
  contributions
• 4:39
  from
• 4:39
  any
• 4:39
  EU
• 4:40
  government
• 4:40
  to
• 4:40
  the
• 4:40
  phpMyAdmin
• 4:41
  open-source
• 4:41
  project,
• 4:42
  despite
• 4:43
  widespread
• 4:43
  dependency
• 4:43
  on
• 4:44
  it.
• 4:44
  Third,
• 4:45
  99%
• 4:46
  of
• 4:46
  European
• 4:46
  governmental
• 4:47
  email
• 4:47
  fails
• 4:48
  current
• 4:48
  encryption
• 4:48
  best
• 4:49
  practices.
• 4:50
  Only
• 4:50
  the
• 4:50
  Netherlands
• 4:50
  (58%
• 4:51
  compliance)
• 4:52
  and
• 4:52
  Denmark
• 4:52
  (44%)
• 4:53
  show
• 4:53
  promising
• 4:53
  numbers.
• 4:54
  There
• 4:54
  are
• 4:54
  no
• 4:55
  EU-level
• 4:55
  TLS
• 4:56
  standards
• 4:56
  for
• 4:56
  government
• 4:57
  email,
• 4:57
  and
• 4:57
  the
• 4:57
  country-specific
• 4:58
  guidelines
• 4:59
  that
• 4:59
  do
• 4:59
  exist
• 4:59
  are
• 4:59
  incompatible.
• 5:01
  The
• 5:01
  platform
• 5:01
  rebuilds
• 5:02
  its
• 5:02
  1,827
• 5:03
  maps
• 5:04
  nightly
• 5:04
  across
• 5:04
  21
• 5:05
  metrics,
• 5:06
  coloring
• 5:06
  each
• 5:06
  region
• 5:06
  traffic-light
• 5:07
  style.
• 5:08
  Right
• 5:08
  now,
• 5:09
  the
• 5:09
  continent
• 5:09
  is
• 5:10
  overwhelmingly
• 5:10
  red.
• 5:13
  The
• 5:13
  gap
• 5:13
  between
• 5:14
  "agents
• 5:14
  that
• 5:14
  work"
• 5:14
  and
• 5:15
  "agents
• 5:15
  you
• 5:15
  would
• 5:15
  trust
• 5:15
  in
• 5:15
  production"
• 5:16
  is
• 5:16
  where
• 5:17
  the
• 5:17
  interesting
• 5:17
  engineering
• 5:18
  happens.
• 5:19
  A
• 5:19
  research
• 5:19
  paper
• 5:19
  from
• 5:19
  Huawei
• 5:20
  Noah's
• 5:20
  Ark
• 5:20
  Lab
• 5:21
  and
• 5:21
  UCL
• 5:22
  pushes
• 5:22
  multi-agent
• 5:23
  coordination
• 5:23
  into
• 5:23
  genuinely
• 5:24
  novel
• 5:24
  territory
• 5:25
  by
• 5:25
  modeling
• 5:25
  it
• 5:25
  as
• 5:26
  organizational
• 5:26
  design.
• 5:27
  OneManCompany
• 5:28
  (OMC)
• 5:28
  separates
• 5:29
  a
• 5:29
  portable
• 5:29
  "Talent"
• 5:30
  identity
• 5:30
  (role,
• 5:30
  skills,
• 5:31
  working
• 5:31
  principles)
• 5:31
  from
• 5:32
  a
• 5:32
  runtime
• 5:32
  "Container"
• 5:32
  (LangGraph,
• 5:33
  Claude
• 5:33
  Code,
• 5:33
  or
• 5:33
  script-based),
• 5:34
  connected
• 5:34
  through
• 5:35
  six
• 5:35
  typed
• 5:35
  interfaces
• 5:36
  that
• 5:36
  mirror
• 5:36
  an
• 5:36
  OS
• 5:36
  kernel:
• 5:37
  process
• 5:37
  management,
• 5:38
  memory,
• 5:38
  filesystem,
• 5:39
  I/O,
• 5:39
  IPC,
• 5:40
  and
• 5:40
  security.
• 5:41
  A
• 5:41
  community-driven
• 5:42
  Talent
• 5:42
  Market
• 5:42
  provides
• 5:42
  on-demand
• 5:43
  recruitment
• 5:43
  through
• 5:43
  three
• 5:44
  channels:
• 5:44
  curated
• 5:45
  open-source
• 5:45
  agents,
• 5:46
  prompt-sourced
• 5:46
  personas
• 5:47
  assembled
• 5:47
  with
• 5:47
  matched
• 5:47
  skills,
• 5:48
  and
• 5:48
  dynamically
• 5:49
  assembled
• 5:49
  agents
• 5:49
  from
• 5:50
  cloud
• 5:50
  skill
• 5:50
  libraries.
• 5:51
  The
• 5:51
  most
• 5:51
  provocative
• 5:52
  element
• 5:52
  is
• 5:52
  OMC's
• 5:53
  HR
• 5:53
  pipeline.
• 5:54
  Agents
• 5:54
  undergo
• 5:55
  one-on-one
• 5:55
  feedback
• 5:56
  after
• 5:56
  each
• 5:56
  task,
• 5:57
  participate
• 5:57
  in
• 5:57
  project
• 5:58
  retrospectives
• 5:58
  that
• 5:58
  produce
• 5:58
  updated
• 5:59
  SOPs,
• 6:00
  and
• 6:00
  face
• 6:00
  formal
• 6:01
  performance
• 6:01
  reviews
• 6:01
  every
• 6:02
  three
• 6:02
  projects.
• 6:03
  Fail
• 6:03
  three
• 6:03
  consecutive
• 6:04
  reviews
• 6:04
  and
• 6:05
  you
• 6:05
  enter
• 6:05
  a
• 6:05
  Performance
• 6:05
  Improvement
• 6:06
  Plan.
• 6:06
  Fail
• 6:07
  one
• 6:07
  more
• 6:07
  and
• 6:07
  it
• 6:07
  is
• 6:07
  automated
• 6:08
  offboarding
• 6:08
  —
• 6:09
  the
• 6:09
  container
• 6:09
  is
• 6:10
  deprovisioned
• 6:10
  and
• 6:10
  the
• 6:10
  capability
• 6:11
  gap
• 6:11
  is
• 6:11
  flagged
• 6:12
  for
• 6:12
  re-recruitment.
• 6:13
  On
• 6:13
  PRDBench
• 6:13
  (50
• 6:14
  project-level
• 6:14
  software
• 6:15
  tasks),
• 6:15
  OMC
• 6:16
  achieved
• 6:16
  84.67%
• 6:17
  success
• 6:18
  rate,
• 6:18
  a
• 6:18
  15-percentage-point
• 6:19
  improvement
• 6:20
  over
• 6:20
  prior
• 6:20
  state
• 6:21
  of
• 6:21
  the
• 6:21
  art,
• 6:21
  at
• 6:21
  about
• 6:21
  $6.91
• 6:22
  per
• 6:23
  task.
• 6:24
  Cross-domain
• 6:24
  case
• 6:24
  studies
• 6:25
  ranged
• 6:25
  from
• 6:25
  $1.57
• 6:27
  for
• 6:27
  audiobook
• 6:27
  production
• 6:28
  to
• 6:28
  $16.26
• 6:29
  for
• 6:29
  an
• 6:29
  automated
• 6:30
  academic
• 6:30
  research
• 6:31
  survey.
• 6:32
  GammaLab's
• 6:32
  Harmonist
• 6:32
  takes
• 6:33
  a
• 6:33
  complementary
• 6:33
  approach:
• 6:34
  instead
• 6:34
  of
• 6:34
  organizational
• 6:35
  metaphors,
• 6:36
  it
• 6:36
  enforces
• 6:37
  protocol
• 6:37
  compliance
• 6:37
  mechanically.
• 6:39
  Every
• 6:39
  code-changing
• 6:40
  turn
• 6:40
  runs
• 6:40
  through
• 6:40
  IDE
• 6:41
  hooks
• 6:41
  that
• 6:41
  verify
• 6:42
  required
• 6:42
  reviewers
• 6:42
  executed,
• 6:43
  memory
• 6:44
  was
• 6:44
  updated,
• 6:45
  and
• 6:45
  protocol
• 6:45
  was
• 6:45
  satisfied
• 6:46
  —
• 6:47
  returning
• 6:47
  a
• 6:47
  structured
• 6:47
  followup
• 6:48
  message
• 6:48
  if
• 6:48
  anything
• 6:49
  is
• 6:49
  missing.
• 6:50
  The
• 6:50
  LLM
• 6:50
  literally
• 6:51
  cannot
• 6:51
  ship
• 6:51
  code
• 6:51
  that
• 6:52
  skipped
• 6:52
  review.
• 6:53
  Supply-chain
• 6:53
  integrity
• 6:54
  is
• 6:54
  baked
• 6:54
  in
• 6:54
  via
• 6:54
  a
• 6:55
  SHA-256
• 6:57
  manifest
• 6:57
  for
• 6:57
  every
• 6:58
  file,
• 6:58
  with
• 6:59
  the
• 6:59
  upgrade
• 6:59
  process
• 6:59
  verifying
• 7:00
  hashes
• 7:00
  before
• 7:01
  copying
• 7:01
  and
• 7:01
  refusing
• 7:02
  tampered
• 7:02
  agent
• 7:02
  definitions
• 7:03
  outright.
• 7:04
  A
• 7:04
  prompt-injection
• 7:05
  scanner
• 7:05
  checks
• 7:05
  agent
• 7:06
  markdown
• 7:06
  for
• 7:06
  override
• 7:07
  attempts,
• 7:07
  exfiltration
• 7:08
  patterns,
• 7:08
  and
• 7:09
  policy
• 7:09
  subversion
• 7:09
  —
• 7:10
  all
• 7:10
  running
• 7:11
  on
• 7:11
  pure
• 7:11
  Python
• 7:11
  stdlib
• 7:12
  and
• 7:12
  bash,
• 7:12
  zero
• 7:13
  external
• 7:13
  dependencies.
• 7:15
  On
• 7:15
  the
• 7:15
  practical
• 7:15
  deployment
• 7:16
  side,
• 7:16
  the
• 7:16
  community
• 7:17
  consensus
• 7:17
  around
• 7:18
  OpenAI's
• 7:18
  Codex-at-scale
• 7:19
  approach
• 7:20
  was
• 7:20
  telling:
• 7:21
  "The
• 7:21
  most
• 7:21
  reliable
• 7:21
  safety
• 7:22
  mechanism
• 7:22
  isn't
• 7:23
  prompt-level
• 7:23
  constraints
• 7:24
  —
• 7:24
  it's
• 7:25
  tool
• 7:25
  scope.
• 7:26
  An
• 7:26
  agent
• 7:26
  that
• 7:26
  can
• 7:26
  only
• 7:27
  write
• 7:27
  to
• 7:27
  specific
• 7:28
  directories
• 7:28
  and
• 7:28
  run
• 7:29
  specific
• 7:29
  commands
• 7:29
  fails
• 7:30
  safely
• 7:30
  when
• 7:31
  it
• 7:31
  tries
• 7:31
  to
• 7:31
  exceed
• 7:31
  scope."
• 7:33
  Hard
• 7:33
  boundaries
• 7:33
  beat
• 7:34
  soft
• 7:34
  instructions
• 7:34
  every
• 7:35
  time.
• 7:36
  DeepSwarm
• 7:36
  offers
• 7:37
  a
• 7:37
  narrower
• 7:37
  but
• 7:37
  practical
• 7:38
  tool:
• 7:39
  task-agnostic
• 7:40
  parallel
• 7:40
  API
• 7:41
  worker
• 7:41
  orchestration
• 7:42
  with
• 7:42
  tiered
• 7:42
  model
• 7:42
  delegation
• 7:43
  (frontier
• 7:43
  model
• 7:44
  plans,
• 7:45
  cheaper
• 7:45
  model
• 7:45
  executes),
• 7:46
  auto-optimized
• 7:47
  worker
• 7:47
  counts,
• 7:48
  and
• 7:48
  99.95%
• 7:50
  API
• 7:50
  success
• 7:51
  rates
• 7:51
  across
• 7:52
  31,000
• 7:53
  calls.
• 7:54
  Daniel
• 7:54
  Miessler's
• 7:54
  PAI
• 7:55
  project
• 7:55
  —
• 7:56
  described
• 7:57
  as
• 7:57
  "an
• 7:57
  OS
• 7:57
  on
• 7:58
  top
• 7:58
  of
• 7:58
  Claude
• 7:58
  Code"
• 7:59
  with
• 7:59
  12,000
• 7:59
  GitHub
• 8:00
  stars
• 8:00
  —
• 8:01
  ripped
• 8:01
  out
• 8:01
  RAG
• 8:02
  entirely
• 8:02
  in
• 8:02
  favor
• 8:03
  of
• 8:03
  plain
• 8:03
  text
• 8:03
  files
• 8:04
  navigated
• 8:04
  with
• 8:04
  ripgrep,
• 8:05
  reasoning
• 8:06
  that
• 8:06
  with
• 8:06
  sufficient
• 8:06
  context
• 8:07
  windows,
• 8:08
  search
• 8:08
  becomes
• 8:08
  navigation.
• 8:10
  Its
• 8:10
  three-tier
• 8:10
  memory
• 8:11
  hierarchy
• 8:11
  (WORK,
• 8:12
  KNOWLEDGE,
• 8:13
  LEARNING)
• 8:13
  compounds
• 8:14
  across
• 8:14
  sessions
• 8:15
  while
• 8:15
  the
• 8:15
  system
• 8:15
  itself
• 8:16
  deliberately
• 8:16
  shrinks
• 8:17
  as
• 8:17
  models
• 8:17
  improve
• 8:17
  —
• 8:18
  what
• 8:18
  the
• 8:19
  developers
• 8:19
  call
• 8:19
  "bitter-pilled
• 8:20
  engineering."
• 8:23
  Mixture-of-Experts
• 8:23
  models
• 8:24
  have
• 8:24
  been
• 8:24
  pitched
• 8:24
  as
• 8:24
  a
• 8:25
  way
• 8:25
  to
• 8:25
  get
• 8:25
  big-model
• 8:26
  performance
• 8:26
  at
• 8:26
  small-model
• 8:27
  cost,
• 8:27
  but
• 8:28
  in
• 8:28
  practice,
• 8:29
  removing
• 8:29
  experts
• 8:29
  kills
• 8:30
  performance
• 8:30
  because
• 8:30
  routing
• 8:31
  learns
• 8:31
  low-level
• 8:32
  lexical
• 8:32
  patterns
• 8:32
  rather
• 8:33
  than
• 8:33
  semantic
• 8:33
  domains.
• 8:34
  Allen
• 8:35
  AI's
• 8:35
  EMO
• 8:35
  (Emergent
• 8:36
  MOdularity)
• 8:36
  fixes
• 8:37
  this
• 8:37
  with
• 8:37
  a
• 8:37
  deceptively
• 8:38
  simple
• 8:38
  change:
• 8:39
  during
• 8:39
  pretraining,
• 8:40
  all
• 8:40
  tokens
• 8:40
  in
• 8:40
  a
• 8:41
  document
• 8:41
  must
• 8:41
  choose
• 8:41
  active
• 8:42
  experts
• 8:42
  from
• 8:42
  a
• 8:42
  shared
• 8:43
  pool.
• 8:44
  This
• 8:44
  single
• 8:45
  constraint
• 8:45
  causes
• 8:45
  expert
• 8:46
  groups
• 8:46
  to
• 8:46
  naturally
• 8:47
  specialize
• 8:47
  by
• 8:47
  domain
• 8:48
  —
• 8:49
  code,
• 8:49
  law,
• 8:50
  biomedical,
• 8:51
  web
• 8:51
  content
• 8:51
  —
• 8:52
  instead
• 8:53
  of
• 8:53
  clustering
• 8:53
  around
• 8:53
  prepositions
• 8:54
  and
• 8:54
  punctuation.
• 8:55
  The
• 8:55
  numbers
• 8:56
  are
• 8:56
  striking.
• 8:57
  EMO
• 8:57
  uses
• 8:57
  1
• 8:58
  billion
• 8:58
  active
• 8:58
  parameters
• 8:59
  from
• 8:59
  14
• 9:00
  billion
• 9:00
  total
• 9:00
  (8
• 9:00
  of
• 9:01
  128
• 9:01
  experts
• 9:02
  per
• 9:02
  token,
• 9:03
  trained
• 9:03
  on
• 9:03
  1
• 9:03
  trillion
• 9:04
  tokens).
• 9:05
  At
• 9:05
  12.5%
• 9:06
  of
• 9:06
  experts
• 9:07
  retained,
• 9:07
  performance
• 9:08
  drops
• 9:08
  only
• 9:08
  about
• 9:09
  3%.
• 9:10
  A
• 9:10
  standard
• 9:10
  MoE
• 9:11
  at
• 9:11
  the
• 9:11
  same
• 9:11
  subset
• 9:11
  size
• 9:12
  degrades
• 9:12
  to
• 9:12
  near-random.
• 9:13
  At
• 9:14
  25%,
• 9:15
  the
• 9:15
  drop
• 9:15
  is
• 9:15
  roughly
• 9:16
  1%.
• 9:17
  Expert
• 9:17
  selection
• 9:18
  is
• 9:18
  remarkably
• 9:18
  cheap
• 9:18
  —
• 9:19
  a
• 9:19
  single
• 9:19
  few-shot
• 9:20
  example
• 9:20
  suffices
• 9:21
  to
• 9:21
  identify
• 9:21
  a
• 9:21
  good
• 9:21
  subset
• 9:22
  for
• 9:22
  any
• 9:22
  domain.
• 9:23
  Global
• 9:24
  load
• 9:24
  balancing
• 9:24
  across
• 9:25
  documents
• 9:25
  rather
• 9:25
  than
• 9:25
  within
• 9:26
  a
• 9:26
  micro-batch
• 9:26
  proved
• 9:27
  essential
• 9:27
  for
• 9:27
  training
• 9:27
  stability,
• 9:28
  and
• 9:28
  the
• 9:29
  pool
• 9:29
  size
• 9:29
  is
• 9:29
  randomly
• 9:30
  sampled
• 9:30
  during
• 9:30
  training
• 9:31
  rather
• 9:31
  than
• 9:31
  fixed.
• 9:32
  The
• 9:32
  practical
• 9:33
  implication
• 9:33
  is
• 9:33
  significant:
• 9:35
  deploy
• 9:35
  only
• 9:35
  the
• 9:35
  expert
• 9:36
  subset
• 9:36
  needed
• 9:36
  for
• 9:36
  a
• 9:36
  specific
• 9:37
  task,
• 9:37
  achieving
• 9:38
  memory-accuracy
• 9:38
  tradeoffs
• 9:39
  standard
• 9:39
  MoEs
• 9:39
  cannot
• 9:40
  offer.
• 9:41
  Model,
• 9:41
  baseline,
• 9:42
  and
• 9:42
  training
• 9:42
  code
• 9:43
  are
• 9:43
  all
• 9:43
  open.
• 9:44
  If
• 9:44
  EMO
• 9:45
  rethinks
• 9:45
  model
• 9:45
  structure,
• 9:46
  Thinking
• 9:47
  Machines
• 9:47
  Lab
• 9:47
  rethinks
• 9:48
  model
• 9:48
  interaction
• 9:48
  entirely.
• 9:49
  Their
• 9:50
  "interaction
• 9:50
  models"
• 9:51
  research
• 9:51
  preview
• 9:51
  replaces
• 9:52
  turn-based
• 9:52
  exchanges
• 9:53
  with
• 9:53
  continuous
• 9:54
  200-millisecond
• 9:54
  micro-turns
• 9:55
  that
• 9:55
  interleave
• 9:56
  input
• 9:56
  processing
• 9:56
  and
• 9:57
  output
• 9:57
  generation
• 9:57
  across
• 9:58
  audio,
• 9:58
  video,
• 9:58
  and
• 9:59
  text
• 9:59
  simultaneously.
• 10:00
  There
• 10:00
  are
• 10:01
  no
• 10:01
  artificial
• 10:01
  turn
• 10:02
  boundaries
• 10:02
  —
• 10:03
  the
• 10:03
  model
• 10:03
  tracks
• 10:03
  implicit
• 10:04
  turn
• 10:04
  states,
• 10:05
  interjects
• 10:05
  proactively,
• 10:06
  translates
• 10:07
  live,
• 10:07
  and
• 10:08
  executes
• 10:08
  concurrent
• 10:08
  tool
• 10:09
  calls
• 10:09
  while
• 10:09
  conversing.
• 10:10
  The
• 10:11
  architecture
• 10:11
  pairs
• 10:11
  a
• 10:12
  real-time
• 10:12
  interaction
• 10:13
  model
• 10:13
  (276B
• 10:14
  MoE,
• 10:15
  12B
• 10:15
  active)
• 10:16
  with
• 10:16
  an
• 10:16
  asynchronous
• 10:17
  background
• 10:17
  model
• 10:18
  for
• 10:18
  deep
• 10:18
  reasoning,
• 10:19
  achieving
• 10:19
  state-of-the-art
• 10:20
  combined
• 10:20
  performance
• 10:21
  on
• 10:21
  both
• 10:21
  intelligence
• 10:22
  and
• 10:22
  interactivity
• 10:23
  benchmarks.
• 10:24
  Existing
• 10:25
  models
• 10:25
  —
• 10:26
  including
• 10:26
  GPT
• 10:27
  Realtime-2
• 10:27
  —
• 10:28
  cannot
• 10:28
  meaningfully
• 10:29
  perform
• 10:29
  the
• 10:29
  visual
• 10:30
  proactivity
• 10:30
  tasks
• 10:31
  the
• 10:31
  team
• 10:31
  introduces,
• 10:32
  such
• 10:32
  as
• 10:32
  counting
• 10:33
  exercise
• 10:33
  repetitions
• 10:34
  or
• 10:34
  responding
• 10:35
  to
• 10:35
  visual
• 10:35
  cues
• 10:35
  without
• 10:36
  being
• 10:36
  prompted.
• 10:40
  A
• 10:40
  single
• 10:40
  command-line
• 10:41
  flag
• 10:41
  is
• 10:41
  saving
• 10:42
  people
• 10:42
  thousands
• 10:42
  of
• 10:42
  dollars.
• 10:43
  A
• 10:43
  llama.cpp
• 10:45
  user
• 10:45
  discovered
• 10:46
  that
• 10:46
  increasing
• 10:46
  the
• 10:46
  micro-batch
• 10:47
  size
• 10:47
  (-ub)
• 10:48
  dramatically
• 10:48
  improves
• 10:49
  prompt
• 10:49
  processing
• 10:49
  for
• 10:50
  partially
• 10:50
  offloaded
• 10:51
  MoE
• 10:51
  models.
• 10:52
  On
• 10:52
  an
• 10:52
  RTX
• 10:52
  3090
• 10:53
  running
• 10:54
  gpt-oss-120b,
• 10:56
  pushing
• 10:56
  -ub
• 10:56
  from
• 10:57
  the
• 10:57
  default
• 10:57
  512
• 10:58
  to
• 10:58
  8192
• 11:00
  boosted
• 11:00
  prefill
• 11:00
  throughput
• 11:01
  from
• 11:01
  380
• 11:02
  tok/s
• 11:03
  to
• 11:03
  2,090
• 11:04
  tok/s
• 11:04
  —
• 11:05
  a
• 11:05
  5.5x
• 11:06
  improvement
• 11:06
  —
• 11:07
  while
• 11:07
  token
• 11:07
  generation
• 11:08
  dropped
• 11:08
  only
• 11:08
  7%.
• 11:10
  The
• 11:10
  trade-off:
• 11:10
  larger
• 11:11
  batches
• 11:11
  need
• 11:11
  more
• 11:11
  GPU
• 11:12
  workspace,
• 11:13
  requiring
• 11:13
  a
• 11:13
  few
• 11:13
  extra
• 11:14
  MoE
• 11:14
  layers
• 11:14
  on
• 11:14
  the
• 11:15
  CPU
• 11:15
  via
• 11:16
  --n-cpu-moe.
• 11:17
  Bigger
• 11:17
  batches
• 11:17
  mean
• 11:18
  fewer
• 11:18
  kernel
• 11:18
  launches,
• 11:19
  keeping
• 11:19
  the
• 11:19
  GPU
• 11:20
  saturated
• 11:20
  during
• 11:21
  prefill,
• 11:21
  while
• 11:22
  generation
• 11:22
  speed
• 11:22
  barely
• 11:23
  changes
• 11:23
  because
• 11:23
  it
• 11:24
  is
• 11:24
  memory-bandwidth
• 11:24
  bound
• 11:25
  on
• 11:25
  CPU
• 11:25
  expert
• 11:26
  weights.
• 11:26
  "One
• 11:26
  of
• 11:27
  the
• 11:27
  reasons
• 11:27
  I
• 11:27
  bought
• 11:27
  a
• 11:27
  DGX
• 11:28
  Spark
• 11:28
  was
• 11:28
  better
• 11:28
  prompt
• 11:29
  processing.
• 11:30
  If
• 11:30
  I
• 11:30
  had
• 11:30
  known
• 11:30
  about
• 11:30
  this
• 11:30
  trick,
• 11:31
  I
• 11:31
  might
• 11:31
  not
• 11:31
  have."
• 11:32
  Speculative
• 11:33
  decoding
• 11:33
  is
• 11:33
  pushing
• 11:34
  consumer
• 11:34
  GPUs
• 11:35
  into
• 11:35
  territory
• 11:35
  that
• 11:35
  previously
• 11:36
  demanded
• 11:36
  multi-card
• 11:37
  setups.
• 11:38
  Gemma
• 11:38
  4
• 11:38
  26B
• 11:39
  (a
• 11:39
  MoE
• 11:40
  with
• 11:40
  4
• 11:40
  billion
• 11:40
  active
• 11:41
  parameters)
• 11:41
  hit
• 11:41
  578
• 11:42
  output
• 11:43
  tokens
• 11:43
  per
• 11:43
  second
• 11:44
  on
• 11:44
  a
• 11:44
  single
• 11:44
  RTX
• 11:45
  5090
• 11:46
  using
• 11:46
  DFlash
• 11:47
  speculative
• 11:47
  decoding
• 11:48
  in
• 11:48
  vLLM
• 11:48
  —
• 11:49
  a
• 11:49
  2.56x
• 11:50
  speedup
• 11:50
  over
• 11:51
  the
• 11:51
  228
• 11:51
  tok/s
• 11:52
  baseline.
• 11:53
  The
• 11:53
  optimal
• 11:54
  setting
• 11:54
  was
• 11:54
  13
• 11:55
  speculative
• 11:55
  tokens
• 11:55
  with
• 11:56
  max
• 11:56
  batched
• 11:56
  tokens
• 11:57
  of
• 11:57
  8192;
• 11:59
  interestingly,
• 12:00
  the
• 12:00
  lowest
• 12:00
  mean
• 12:00
  latency
• 12:01
  configuration
• 12:02
  was
• 12:02
  not
• 12:02
  the
• 12:02
  best
• 12:02
  serving
• 12:03
  configuration
• 12:03
  due
• 12:04
  to
• 12:04
  worse
• 12:04
  p95
• 12:05
  tail
• 12:05
  latency.
• 12:06
  Community
• 12:06
  pushback
• 12:07
  noted
• 12:07
  DFlash
• 12:08
  performance
• 12:08
  degrades
• 12:09
  above
• 12:09
  20K
• 12:09
  context
• 12:10
  and
• 12:10
  that
• 12:10
  random
• 12:11
  dataset
• 12:11
  benchmarks
• 12:12
  overstate
• 12:12
  acceptance
• 12:13
  rates
• 12:13
  versus
• 12:13
  real
• 12:14
  workloads
• 12:14
  —
• 12:15
  but
• 12:15
  for
• 12:15
  short-context
• 12:16
  inference,
• 12:16
  the
• 12:16
  throughput
• 12:17
  is
• 12:17
  undeniable.
• 12:19
  AMD's
• 12:19
  viability
• 12:20
  as
• 12:20
  a
• 12:20
  full
• 12:20
  training
• 12:21
  platform
• 12:21
  gets
• 12:22
  a
• 12:22
  concrete
• 12:22
  data
• 12:22
  point
• 12:23
  with
• 12:23
  ZAYA1-8B,
• 12:26
  a
• 12:26
  frontier
• 12:27
  MoE
• 12:27
  model
• 12:27
  pretrained
• 12:28
  entirely
• 12:28
  on
• 12:28
  1,024
• 12:29
  AMD
• 12:30
  MI300X
• 12:33
  nodes
• 12:33
  with
• 12:33
  Pensando
• 12:34
  Pollara
• 12:34
  interconnect.
• 12:36
  The
• 12:36
  architecture
• 12:36
  introduces
• 12:37
  Markovian
• 12:37
  RSA
• 12:38
  (a
• 12:38
  novel
• 12:38
  reinforcement
• 12:39
  learning
• 12:39
  technique)
• 12:40
  and
• 12:40
  uses
• 12:40
  0.8
• 12:41
  billion
• 12:41
  active
• 12:42
  parameters
• 12:42
  from
• 12:42
  8
• 12:43
  billion
• 12:43
  total.
• 12:44
  Whether
• 12:44
  it
• 12:45
  beats
• 12:45
  Qwen-3.5-9B
• 12:46
  is
• 12:46
  debatable,
• 12:47
  but
• 12:48
  the
• 12:48
  training
• 12:48
  stack
• 12:48
  proof-of-concept
• 12:49
  matters:
• 12:50
  "The
• 12:50
  hardest
• 12:51
  part
• 12:51
  is
• 12:51
  always
• 12:51
  the
• 12:52
  first
• 12:52
  run
• 12:52
  for
• 12:52
  a
• 12:52
  new
• 12:52
  lab.
• 12:53
  And
• 12:53
  given
• 12:54
  that
• 12:54
  they're
• 12:54
  running
• 12:54
  on
• 12:54
  an
• 12:54
  AMD
• 12:55
  stack,
• 12:55
  they
• 12:56
  had
• 12:56
  an
• 12:56
  even
• 12:56
  bigger
• 12:56
  hill
• 12:56
  to
• 12:57
  climb."
• 12:58
  On
• 12:58
  the
• 12:58
  tooling
• 12:58
  front,
• 12:59
  ggerganov
• 12:59
  merged
• 13:00
  llama-eval
• 13:00
  directly
• 13:01
  into
• 13:01
  llama.cpp,
• 13:03
  providing
• 13:03
  standardized
• 13:04
  evaluation
• 13:04
  against
• 13:04
  AIME,
• 13:05
  AIME2025,
• 13:07
  GSM8K,
• 13:09
  and
• 13:10
  GPQA
• 13:10
  without
• 13:11
  the
• 13:11
  API-key-or-transformers
• 13:12
  dependency
• 13:13
  that
• 13:13
  plagues
• 13:13
  most
• 13:13
  benchmarking
• 13:14
  setups
• 13:14
  —
• 13:15
  "saves
• 13:16
  everyone
• 13:16
  from
• 13:16
  setting
• 13:16
  up
• 13:16
  their
• 13:17
  own
• 13:17
  janky
• 13:17
  benchmark
• 13:18
  pipeline
• 13:18
  that
• 13:18
  measures
• 13:18
  different
• 13:19
  things."
• 13:20
  And
• 13:20
  at
• 13:20
  the
• 13:20
  opposite
• 13:20
  extreme,
• 13:21
  someone
• 13:21
  got
• 13:22
  Karpathy's
• 13:22
  TinyStories-260K
• 13:24
  running
• 13:24
  on
• 13:24
  a
• 13:24
  stock
• 13:24
  Game
• 13:24
  Boy
• 13:25
  Color
• 13:25
  —
• 13:26
  INT8
• 13:26
  weights,
• 13:27
  fixed-point
• 13:27
  math,
• 13:28
  bank-switched
• 13:28
  cartridge
• 13:29
  ROM,
• 13:29
  KV
• 13:29
  cache
• 13:30
  in
• 13:30
  SRAM.
• 13:31
  It
• 13:31
  is
• 13:31
  extremely
• 13:32
  slow
• 13:32
  and
• 13:32
  the
• 13:32
  output
• 13:33
  is
• 13:33
  gibberish,
• 13:34
  but
• 13:34
  transformer
• 13:34
  prefill
• 13:35
  and
• 13:35
  autoregressive
• 13:36
  generation
• 13:36
  genuinely
• 13:37
  execute
• 13:37
  on
• 13:37
  1998
• 13:38
  hardware
• 13:38
  with
• 13:38
  32KB
• 13:39
  of
• 13:39
  work
• 13:39
  RAM.
• 13:41
  As
• 13:41
  one
• 13:41
  commenter
• 13:41
  put
• 13:42
  it:
• 13:42
  "Pointless.
• 13:43
  Therefore,
• 13:44
  indispensable."
• 13:47
  A
• 13:48
  new
• 13:48
  benchmark
• 13:48
  called
• 13:49
  IntentGrasp
• 13:49
  puts
• 13:49
  numbers
• 13:50
  on
• 13:50
  something
• 13:50
  practitioners
• 13:51
  have
• 13:51
  long
• 13:51
  suspected:
• 13:52
  AI
• 13:53
  models
• 13:53
  are
• 13:53
  substantially
• 13:54
  worse
• 13:54
  at
• 13:54
  understanding
• 13:55
  what
• 13:55
  users
• 13:55
  actually
• 13:55
  want
• 13:56
  than
• 13:56
  at
• 13:56
  generating
• 13:56
  fluent
• 13:57
  responses.
• 13:58
  Across
• 13:58
  20
• 13:59
  models
• 13:59
  from
• 13:59
  seven
• 13:59
  families,
• 14:00
  all
• 14:00
  scored
• 14:00
  below
• 14:01
  60%
• 14:01
  on
• 14:01
  the
• 14:02
  full
• 14:02
  benchmark
• 14:02
  and
• 14:02
  below
• 14:03
  25%
• 14:04
  on
• 14:04
  the
• 14:04
  harder
• 14:04
  "Gem
• 14:04
  Set."
• 14:05
  The
• 14:05
  paper
• 14:06
  distinguishes
• 14:06
  declared
• 14:07
  intent
• 14:07
  (what
• 14:07
  someone
• 14:07
  types)
• 14:08
  from
• 14:08
  enacted
• 14:08
  intent
• 14:08
  (what
• 14:09
  someone
• 14:09
  does),
• 14:10
  arguing
• 14:10
  that
• 14:10
  as
• 14:10
  agents
• 14:11
  turn
• 14:11
  language
• 14:11
  into
• 14:11
  action,
• 14:12
  misinterpretation
• 14:13
  does
• 14:13
  not
• 14:13
  produce
• 14:14
  a
• 14:14
  wrong
• 14:14
  answer
• 14:14
  —
• 14:15
  it
• 14:15
  produces
• 14:15
  a
• 14:15
  correctly
• 14:16
  executed
• 14:16
  wrong
• 14:16
  goal.
• 14:17
  The
• 14:17
  good
• 14:18
  news:
• 14:18
  intentional
• 14:19
  fine-tuning
• 14:19
  substantially
• 14:20
  improves
• 14:20
  performance,
• 14:21
  meaning
• 14:21
  intent
• 14:21
  comprehension
• 14:22
  is
• 14:22
  trainable,
• 14:23
  not
• 14:23
  a
• 14:23
  fixed
• 14:23
  ceiling.
• 14:24
  The
• 14:24
  commercial
• 14:25
  implications
• 14:25
  are
• 14:25
  clear:
• 14:26
  organizations
• 14:27
  wanting
• 14:27
  agents
• 14:27
  to
• 14:27
  handle
• 14:27
  more
• 14:28
  valuable
• 14:28
  work
• 14:28
  need
• 14:28
  better
• 14:29
  intent
• 14:29
  interpretation
• 14:30
  before
• 14:30
  they
• 14:30
  widen
• 14:30
  authority
• 14:30
  —
• 14:31
  "clarification
• 14:32
  loops,
• 14:32
  context
• 14:32
  grounding,
• 14:33
  task
• 14:33
  boundaries,
• 14:34
  escalation
• 14:34
  points,
• 14:34
  and
• 14:35
  stop
• 14:35
  conditions."
• 14:36
  The
• 14:36
  evaluation
• 14:36
  problem
• 14:37
  runs
• 14:37
  deeper
• 14:37
  than
• 14:37
  intent.
• 14:38
  A
• 14:38
  candid
• 14:38
  post
• 14:39
  on
• 14:39
  r/learnmachinelearning
• 14:40
  confesses
• 14:40
  what
• 14:40
  many
• 14:41
  teams
• 14:41
  will
• 14:41
  not:
• 14:41
  "Most
• 14:42
  of
• 14:42
  what
• 14:42
  I've
• 14:42
  been
• 14:42
  calling
• 14:42
  'evals'
• 14:43
  has
• 14:43
  been
• 14:43
  vibe
• 14:43
  checks."
• 14:44
  The
• 14:44
  author
• 14:45
  nearly
• 14:45
  shipped
• 14:45
  Haiku
• 14:46
  over
• 14:46
  Sonnet
• 14:46
  based
• 14:46
  on
• 14:47
  testing
• 14:47
  six
• 14:47
  prompts,
• 14:48
  only
• 14:48
  to
• 14:48
  find
• 14:48
  in
• 14:49
  a
• 14:49
  proper
• 14:49
  held-out
• 14:49
  evaluation
• 14:50
  that
• 14:50
  Sonnet
• 14:50
  was
• 14:50
  winning
• 14:51
  by
• 14:51
  a
• 14:51
  meaningful
• 14:51
  margin
• 14:51
  on
• 14:52
  the
• 14:52
  cases
• 14:52
  that
• 14:52
  mattered
• 14:52
  most.
• 14:53
  A
• 14:54
  real
• 14:54
  eval
• 14:54
  requires
• 14:55
  a
• 14:55
  held-out
• 14:55
  dataset,
• 14:56
  a
• 14:56
  scoring
• 14:56
  rubric,
• 14:56
  and
• 14:57
  a
• 14:57
  comparison
• 14:57
  table
• 14:57
  —
• 14:58
  "when
• 14:59
  an
• 14:59
  engineer
• 14:59
  asks
• 14:59
  'why
• 15:00
  are
• 15:00
  we
• 15:00
  picking
• 15:00
  this'
• 15:00
  you
• 15:01
  have
• 15:01
  numbers
• 15:01
  not
• 15:01
  vibes."
• 15:02
  The
• 15:03
  community
• 15:03
  response
• 15:03
  confirmed
• 15:04
  how
• 15:04
  endemic
• 15:04
  the
• 15:04
  problem
• 15:05
  is:
• 15:05
  "6
• 15:06
  prompts
• 15:06
  and
• 15:06
  a
• 15:06
  gut
• 15:07
  feeling
• 15:07
  being
• 15:07
  called
• 15:07
  evals
• 15:08
  is
• 15:08
  way
• 15:08
  more
• 15:08
  common
• 15:08
  than
• 15:09
  people
• 15:09
  want
• 15:09
  to
• 15:09
  admit."
• 15:10
  Claude
• 15:10
  specifically
• 15:11
  is
• 15:11
  developing
• 15:12
  what
• 15:12
  users
• 15:12
  describe
• 15:13
  as
• 15:13
  "radical
• 15:13
  over-honesty
• 15:14
  mixed
• 15:14
  with
• 15:14
  absolute
• 15:15
  certainty
• 15:15
  about
• 15:15
  timelines
• 15:16
  it
• 15:16
  doesn't
• 15:16
  understand"
• 15:17
  —
• 15:18
  confidently
• 15:18
  estimating
• 15:18
  months
• 15:19
  of
• 15:19
  team
• 15:19
  effort
• 15:19
  for
• 15:19
  things
• 15:20
  already
• 15:20
  running
• 15:20
  in
• 15:20
  another
• 15:21
  terminal.
• 15:22
  The
• 15:22
  training
• 15:22
  data
• 15:22
  still
• 15:23
  assumes
• 15:23
  software
• 15:24
  is
• 15:24
  built
• 15:24
  through
• 15:24
  committees
• 15:24
  and
• 15:25
  sprint
• 15:25
  planning;
• 15:26
  meanwhile
• 15:26
  practitioners
• 15:27
  are
• 15:27
  shipping
• 15:27
  entire
• 15:28
  frameworks
• 15:28
  between
• 15:29
  breakfast
• 15:29
  and
• 15:29
  dinner.
• 15:30
  The
• 15:30
  irony:
• 15:31
  the
• 15:31
  same
• 15:31
  model
• 15:31
  lecturing
• 15:32
  about
• 15:32
  ambition
• 15:33
  is
• 15:33
  often
• 15:33
  fully
• 15:34
  capable
• 15:34
  of
• 15:34
  building
• 15:34
  the
• 15:35
  thing
• 15:35
  if
• 15:35
  you
• 15:35
  ignore
• 15:35
  the
• 15:35
  lecture
• 15:36
  and
• 15:36
  keep
• 15:36
  prompting.
• 15:38
  The
• 15:38
  practical
• 15:38
  remedy,
• 15:39
  according
• 15:39
  to
• 15:39
  Ofer
• 15:39
  Maor,
• 15:40
  is
• 15:40
  structured
• 15:41
  team-wide
• 15:41
  AI
• 15:42
  literacy:
• 15:42
  weekly
• 15:43
  show-and-tell
• 15:43
  sessions
• 15:44
  where
• 15:44
  people
• 15:44
  demonstrate
• 15:45
  what
• 15:45
  they
• 15:45
  have
• 15:45
  built,
• 15:46
  pushing
• 15:46
  everyone
• 15:47
  past
• 15:47
  the
• 15:47
  "event
• 15:48
  horizon"
• 15:48
  of
• 15:48
  AI
• 15:48
  savviness
• 15:49
  where
• 15:49
  it
• 15:49
  becomes
• 15:50
  hard
• 15:50
  not
• 15:50
  to
• 15:50
  be
• 15:51
  a
• 15:51
  fanatic.
• 15:52
  On
• 15:52
  the
• 15:52
  infrastructure
• 15:52
  side,
• 15:53
  FaultLine
• 15:53
  tackles
• 15:54
  LLM
• 15:54
  memory
• 15:54
  reliability
• 15:55
  by
• 15:56
  building
• 15:56
  a
• 15:56
  searchable
• 15:56
  knowledge
• 15:57
  graph
• 15:57
  from
• 15:57
  conversations
• 15:58
  with
• 15:58
  explicit
• 15:58
  validation
• 15:59
  gates.
• 16:00
  Facts
• 16:00
  are
• 16:00
  validated
• 16:01
  against
• 16:01
  existing
• 16:01
  knowledge,
• 16:02
  stored
• 16:03
  in
• 16:03
  a
• 16:03
  user-controlled
• 16:03
  database,
• 16:04
  and
• 16:04
  injected
• 16:05
  as
• 16:05
  grounding
• 16:05
  context
• 16:06
  before
• 16:06
  the
• 16:06
  model
• 16:07
  responds.
• 16:08
  Short-term
• 16:08
  vector
• 16:09
  search
• 16:09
  handles
• 16:09
  recent
• 16:10
  context
• 16:10
  while
• 16:10
  long-term
• 16:11
  relational
• 16:11
  storage
• 16:12
  locks
• 16:12
  in
• 16:12
  validated
• 16:12
  facts
• 16:13
  across
• 16:13
  sessions.
• 16:14
  The
• 16:14
  project
• 16:15
  draws
• 16:15
  on
• 16:15
  several
• 16:16
  open
• 16:16
  research
• 16:16
  problems
• 16:17
  —
• 16:18
  write
• 16:18
  gates,
• 16:18
  fact
• 16:19
  lifecycle
• 16:19
  management,
• 16:20
  self-building
• 16:21
  ontologies
• 16:21
  —
• 16:22
  that
• 16:22
  remain
• 16:22
  unshipped
• 16:23
  in
• 16:23
  production
• 16:23
  systems,
• 16:24
  making
• 16:24
  it
• 16:25
  an
• 16:25
  early
• 16:25
  test
• 16:25
  of
• 16:25
  whether
• 16:26
  academic
• 16:26
  memory
• 16:26
  governance
• 16:27
  concepts
• 16:27
  actually
• 16:28
  work
• 16:28
  outside
• 16:28
  the
• 16:28
  lab.