Latest / Agentic Conversations (formally mlops.community) / Coding Agents Are Secretly General Agents

Transcript

Coding Agents Are Secretly General Agents · Agentic Conversations (formally mlops.community)

• 0:23
  Fuck, were you even doing in 2008?
• 0:25
  So in computer vision, it was SIFT features and SVMS that
• 0:28
  that's basically the state-of-the-art at the time.
• 0:30
  There were a couple other things that people are using, but.
• 0:32
  Not even using GPU's. I think that some there was,
• 0:37
  yeah, basically Alex was the first one to to write
• 0:40
  specialized kernels. There was some applications, I'm
• 0:44
  sure where it like, you know, optical flow, it's highly
• 0:47
  paralyzable. And I wouldn't be surprised if
• 0:49
  there were some labs that were doing things along those lines.
• 0:51
  But it definitely was not like a scaling era for machine
• 0:54
  learning. Largest data sets you know you
• 0:56
  could fit on a USB stick. What gave you the inspiration to
• 1:00
  start Code Jet? And Like, you must have seen
• 1:05
  something that was like, oh, this is getting better.
• 1:08
  We could probably do something here because there was a company
• 1:10
  that tried it and then they folded, right?
• 1:12
  I can't remember the name of them.
• 1:14
  The history of cogeneration is littered with companies that
• 1:16
  were a little too early but like, totally visionary.
• 1:19
  So you're probably thinking of Tab 9.
• 1:20
  I think Tab 9's actually still around, and they're still.
• 1:22
  Going and then it wasn't them. There was one that height I
• 1:26
  think it was kite in like 2022 right?
• 1:28
  Or 2020 maybe? Maybe even before that a little.
• 1:32
  Bit like 2021 right? They folded right when Chachi BT
• 1:37
  started getting really popular and it was very confusing to me
• 1:40
  because I'm like wait GitHub copilot is printing money?
• 1:44
  Supposedly they just hit like a billion in revenue after they
• 1:48
  released and kite shut down. You hate to see it.
• 1:53
  I mean, you know, any start up dying.
• 1:55
  It's sad to see. I think that one of my learnings
• 1:58
  from doing startups for about a decade is that momentum begets
• 2:01
  momentum. And if you are too early, you
• 2:04
  know, you kind of get people all hyped up on this vision and then
• 2:07
  you're actually not the player who ends up bringing it to
• 2:09
  market. Maybe some people can kind of
• 2:10
  catch on and they end up taking off at that point.
• 2:12
  But in the majority of cases it ends up being that.
• 2:15
  Yeah, get up. Copilot is the one who
• 2:16
  introduced the concept. That's what people get excited
• 2:18
  about. That's what gets hyped up on
• 2:19
  Twitter. Kite is remembered as like the,
• 2:22
  you know, everybody tried it in 2021 or something like that when
• 2:25
  Paul Graham hyped it up and they're like, this isn't really
• 2:28
  ready for production yet. And get up Copilot was, you
• 2:31
  know, is directly integrated into VS Code from the very get
• 2:33
  go. And so you don't really have to
• 2:34
  do very much in order to get access to it.
• 2:37
  But then the sort of I think the bigger revelation was cursor,
• 2:41
  Cursor get up Copilot was 2022, I want to say is when that came
• 2:44
  out and it immediately it was in the happy path, which was
• 2:48
  amazing. So you don't really have to do
• 2:50
  very much in order to benefit from it, right?
• 2:51
  It would just do auto completions for you once you
• 2:53
  flipped it on. And I think that that same
• 2:55
  experience of basically not having to take any additional
• 2:58
  action, but just having AI seamlessly injected into your
• 3:01
  workflow that's going to roll out to a bunch of other
• 3:02
  industries very soon. I think that cursor is sort of
• 3:07
  the, the thing about cursor was nobody could tell you not to
• 3:10
  download it. So like most of the people who
• 3:12
  ended up adopting it early on, these sort of hype guys on, on
• 3:14
  Twitter, which, you know, I'm very much so a part of that
• 3:17
  ecosystem where people who are like working at, you know, I
• 3:20
  don't know, JP Morgan or something like that.
• 3:22
  They download this editor, they swipe their personal credit card
• 3:25
  for 20 bucks a month. And then for me, the revelation
• 3:27
  was the composer mode where as a sidebar that pops out and you
• 3:29
  can kind of chat with it really kind of hit stride with Claude
• 3:32
  Sonic 3.5, which was probably mid 2024, something like that.
• 3:36
  Yeah, that's the inflection point right there.
• 3:39
  But then code, Jen, what made you want to start it?
• 3:43
  And did you see that? How did you see the writing on
• 3:45
  the wall? Depends how far you want to go
• 3:49
  back, I would say, you know, I've always been interested in
• 3:51
  artificial intelligence. You know, I think that the idea
• 3:54
  of the machine writing itself and creating itself is this sort
• 3:57
  of recursive building and something that's always really
• 3:58
  appealed to me going way back to, I mean, even high school.
• 4:01
  Is the book good to Lecher Bach? You might be familiar with it
• 4:04
  sort of like a It's a Pulitzer Prize winning book and it's.
• 4:06
  Way overestimate my culture. Got it.
• 4:11
  I do not know that at all. What is?
• 4:13
  It, it's a book called Gerdelescherbach.
• 4:14
  It's by this guy Doug Hofstadter, who he sort of like
• 4:18
  invented the Simsys major at Stanford in a way, or at least
• 4:21
  the the major that Marissa Mayer and many other people sort of
• 4:24
  made famous. It's like based on this book
• 4:26
  almost or directly reflects the contents of the book.
• 4:28
  At least it sort of coincided with him writing this book went
• 4:32
  on to win the Pulitzer Prize. The core concept of it is you
• 4:35
  have MC Escher, the painter, who you might know the painting of
• 4:38
  him. Yeah, exactly.
• 4:40
  Johann Smaschenbach, the musician.
• 4:42
  And then Kurt Godel, I'm definitely mispronouncing his
• 4:44
  name, but he's a logician. And all their works have some
• 4:48
  inherent properties to them or substructure that's shared.
• 4:51
  And usually that has to do with self reference.
• 4:54
  So for example, there are exactly the stairs, the, you
• 4:57
  know, hands writing themselves. And there's actually like a lot
• 4:59
  of like really interesting mathematical concepts embedded
• 5:02
  both in Bach's work, whether he meant it or not, and MC Escher.
• 5:06
  And then Kurt Godel is famous for the essential incompleteness
• 5:09
  theorem, which is basically this idea that any sufficiently
• 5:12
  powerful logical system will necessarily end up containing
• 5:16
  things that can neither be true, proved that are true, but cannot
• 5:19
  be proven to be true. And the way in which it does
• 5:22
  that is basically it contains something that looks like the
• 5:25
  epimetitis paradox, which is where it's the guy gets up in
• 5:28
  front of, you know, a, a bunch of people up amenities is the
• 5:31
  guy gets up in front of a bunch of people in Rome or in Greece
• 5:33
  and he says, citizens of Athens, I am lying to you.
• 5:37
  And because the statement refers to itself, then it's the self
• 5:42
  reference is basically what enables it to be contradictory.
• 5:44
  So it's a paradox. And any sufficiently powerful
• 5:47
  number system can contain a self reference like that that enables
• 5:51
  it to basically say this statement is not provable.
• 5:54
  And so anyways, it's a little bit of a sidetrack here, but the
• 5:58
  interesting part about this book is it ties together all three of
• 6:00
  these different concepts or these different, you know, you
• 6:04
  know, prolific people's works, MC Escher, JS Bach and Kurt
• 6:07
  Godel. And basically it's about
• 6:09
  entities that contain self reference.
• 6:11
  And the guy uses that as a jumping off point to talk about
• 6:13
  AI. And eventually he talks about
• 6:15
  the emergence of consciousness comes from systems that
• 6:18
  represent the outside world. And part of the representation
• 6:21
  of the outside world in this symbolic system contains
• 6:24
  representation of self. There you go.
• 6:26
  You landed the plane beautifully.
• 6:28
  There we go. And so code Gen. from that book.
• 6:34
  So anyways that that that sort of kicked me off.
• 6:36
  I'm getting really interested in the stuff.
• 6:38
  So Jenny and I went through. I worked at Palantir, I studied
• 6:41
  AI in college, I did a bunch of machine learning stuff to the
• 6:44
  decades worth of startups, give or take.
• 6:47
  And around 2020 is when. So I've always kept up with the
• 6:52
  state-of-the-art of yeah, research.
• 6:53
  It's just sort of something that's very interesting to me.
• 6:55
  2020 is when the GBT 3 paper came out, which was essentially
• 6:58
  just scaling up language models and then ended up having this
• 7:01
  incredible property. So basically few shot learning.
• 7:03
  So you can give it pretty much any problem that we were dealing
• 7:05
  with in NLP at the time. It's just like you few shot
• 7:08
  prompt this thing and bam, it's better than whatever you had in
• 7:10
  mind. And so I knew this was going to
• 7:12
  be big. I remember joking with somebody
• 7:14
  that like between Bitcoin and GBT three, it wasn't clear which
• 7:17
  would have a bigger impact on the world and they were like,
• 7:19
  that's so no, like obviously Bitcoin has a bigger impact and
• 7:23
  I guess TBD. 2020, yeah. 2020 it wasn't clear right?
• 7:27
  And then 2022 is when the API opened up from Open AI and I had
• 7:32
  just basically sold my previous company.
• 7:34
  I had ended up my lock up period and I ended up leaving for
• 7:36
  various reasons. And I entered into this sort of
• 7:38
  flow state of doing what I called one demo per week where I
• 7:41
  would literally just sit on my computer and come up with really
• 7:44
  interesting concepts of something you can implement
• 7:46
  using the GPT 3 API and some image generation stuff as well.
• 7:50
  And just put it on Twitter and see what people think and try
• 7:53
  and get, you know, people excited about it.
• 7:54
  And so couple notable ones from that era for me, where I think I
• 7:58
  was the first person to ship text to Figma.
• 8:00
  And so you could input a prompt and you could say, hey, give me
• 8:02
  an iPhone screen with a bunch of other stuff on it and it would
• 8:06
  essentially one shot it and generate you a schema file.
• 8:09
  And then that would render to Figma.
• 8:11
  I did a text to auto tool or sorry, text to retool, but it's
• 8:16
  actually known on retools, like on an open source one where you
• 8:18
  could describe a dashboard and it would have data schema that
• 8:21
  was referencing and it would essentially create a data
• 8:24
  dashboard for you. And so, you know, what I sort of
• 8:27
  discovered over the process of building a bunch of these
• 8:29
  applications is that the things that made AL one application
• 8:34
  successful were verifiability built into the domain.
• 8:37
  And so anytime you're doing some type of a schema generation like
• 8:40
  that, you can essentially lint it.
• 8:41
  You can go through and say, is this a valid schema?
• 8:43
  Does it compile? You know, they're they're like
• 8:46
  specific empirical things you can look for.
• 8:48
  And if it's wrong, you can actually detect that error and
• 8:50
  basically pass it back to the model as a residual and say
• 8:53
  figure it out dude. Which is so clear now, but in
• 8:58
  those days that's incredible. You probably felt like you
• 9:01
  stumbled on gold. I think that I don't want to
• 9:05
  give myself too, too much credit here, like I think a lot of
• 9:07
  other people were, you know, also seeing like, OK, it was
• 9:09
  limited value and like an e-mail thing for you.
• 9:12
  Yeah. Summarization or.
• 9:14
  Yeah, auto complete or something like that.
• 9:15
  But anybody who's coding, you know, with compile at the time
• 9:18
  was like, wow, this thing is incredible.
• 9:19
  And you know, it's actually, there's so much low hanging
• 9:22
  fruit in code too, where it's just boilerplate.
• 9:24
  Like you don't really benefit from writing a Java class
• 9:27
  implementation except for a couple lines of code in there.
• 9:30
  So it's very well suited. And it compiles, which is
• 9:33
  different than a lot of the other things that you have to
• 9:36
  eval. Exactly.
• 9:37
  Yeah. So, you know, and the
• 9:39
  implications of that are pretty clear as well with respect to
• 9:41
  our LVR reinforcement learning from verifiable rewards, where
• 9:45
  you can have an agent go off and write code and try and pass the
• 9:47
  unit test. If it doesn't pass the unit
• 9:49
  test, you say here's the failure, fix it.
• 9:51
  And then eventually it goes off and it does pass the unit test.
• 9:54
  And then you say, OK, learn from your mistakes.
• 9:57
  Or maybe you rephrase your trajectory so it's perfect and
• 10:00
  then you fine tune on that. I remember seeing one paper
• 10:04
  actually, that was really an aha moment where they were trying to
• 10:07
  train, I think it was for performance improvements.
• 10:11
  So they said, OK, here's a certain function that we have
• 10:14
  and we want to train a model to be good at rewriting the
• 10:17
  function such that it's more performant.
• 10:19
  You can run the function and you can measure the wall time it
• 10:21
  takes to actually the number of CBU cycles it takes to run a
• 10:24
  function. And so they'd have a try, you
• 10:25
  know, 100 different variants of it.
• 10:27
  The one that actually ended up being more performant, they'd
• 10:29
  say, OK, this is the one that we're going to end up fine
• 10:31
  tuning on. And that's now we just call that
• 10:32
  RLVR. But this is one of the first
• 10:34
  applications to that. And so it was also pretty clear.
• 10:38
  I guess the point I'm trying to make is that the direction that
• 10:41
  research would go in would be these things are going to get
• 10:43
  really good at coding really fast because the feedback loop
• 10:47
  is instantaneous. If you're working at Anthropic,
• 10:49
  like you're writing code with these things all day, you can
• 10:51
  figure out what it's lacking and then that informs the type of,
• 10:54
  you know, emails you put together.
• 10:55
  Yeah, all the necessary like boundary conditions are set up
• 10:59
  for your success. Exactly.
• 11:02
  There's so much data you can produce it synthetically.
• 11:05
  And at the time, in 2022, people like Gary Marcus were getting
• 11:08
  out there and saying like. We're running out of data.
• 11:11
  Which everybody knew was, you know, first, it's false that we
• 11:14
  didn't run out of data. There's still plenty of, you
• 11:16
  know, appreciating data available.
• 11:17
  But also it completely ignores the fact that you can
• 11:20
  synthetically create. There's synthetic data, and the
• 11:25
  Internet is doubled like every three days, the amount of data
• 11:30
  that's on there. I know that a lot of the new
• 11:32
  data is coming from LMS, but this conversation is going to go
• 11:36
  out. And it wasn't on the Internet
• 11:39
  yesterday. Yeah.
• 11:41
  So like that kind of stuff. Yeah.
• 11:44
  I was always a little bit skeptical on that idea of we're
• 11:47
  running out of data. But then you had ILIA came out
• 11:50
  and said it had some nerve apps too.
• 11:52
  And it's like this guy's way in a different league of brain
• 11:58
  power than me. Like, should I question my
• 12:01
  belief? So I don't know.
• 12:04
  Yeah, the the Internet doubling is probably increasingly not
• 12:07
  from human generated data. So you have to be a little bit
• 12:09
  suss of the stuff. There's the sexually Speaking of
• 12:13
  ML off one of my favorite papers of all time is it's called
• 12:17
  machine learning, the highest interest credit card.
• 12:20
  The one you had to reference, any talk that was given around
• 12:24
  machine learning in ML OPS and from like 2020 to 2022, it was
• 12:29
  you had to reference that box of like the models.
• 12:32
  Just a small piece of this, right?
• 12:35
  And there's so many opportunities to shoot yourself
• 12:36
  in the foot. And the example that sticks with
• 12:40
  me from that paper is they talk about training Google Translate
• 12:44
  and how, you know, they assemble parallel corpora.
• 12:47
  You get a bunch of English pages, you get the Spanish
• 12:49
  translation when they're, you know, available in both.
• 12:51
  And then you train it something to map between the two.
• 12:53
  And then they realize at a certain point that actually most
• 12:56
  of the Spanish pages on the Internet are the output of
• 12:58
  Google Translate. And so they're essentially
• 13:00
  drinking their own backwash. And I think that that's
• 13:03
  absolutely what would happen if you just try to pre train on all
• 13:06
  text on the, you know, on Reddit 2026 onward.
• 13:09
  It's like probably mostly open claw at this point or whatever
• 13:12
  he calls it. Yeah.
• 13:13
  So you gotta be a little careful.
• 13:14
  Yeah. And it's essentially getting
• 13:16
  these loops of like confirming to itself that the correct
• 13:19
  translation for an idiom is, you know, whatever it thought was
• 13:22
  like collapses in weird ways. Exactly.
• 13:25
  Now then you just sold Cogen to click up.
• 13:29
  Walk me through maybe the process of selling and then also
• 13:34
  now what you're doing. For sure.
• 13:37
  Yeah. So we built Cogen over three
• 13:38
  years, the original concept behind it, we started it when I
• 13:42
  think Cursor had already kind of hit their strut.
• 13:44
  So it's clear that they were going to win.
• 13:45
  I think if you asked me, like in early 2022 or sorry, early 2023,
• 13:50
  what's the landscape going to look like?
• 13:51
  I'd say Cursor, this can be the editor of choice.
• 13:54
  Maybe the S code will make a comeback.
• 13:55
  Didn't happen, You know, maybe there'll be like one or two
• 13:58
  other players. And so we thought very early on,
• 14:01
  OK, we're going to essentially leapfrog and we're going to
• 14:03
  shoot to where the puck is going, schedule where the puck
• 14:05
  is going and build a fully autonomous background agent.
• 14:08
  And so the idea was ticket to pull request.
• 14:10
  You essentially create a linear ticket or click up ticket or
• 14:13
  something like that would say, I want this change that.
• 14:15
  It would spin up an agent in a sandbox who would go off and
• 14:18
  write the code and then submit a pull request back.
• 14:21
  So you wouldn't even have to deal with the ID.
• 14:23
  No ID. You should be able to pull it
• 14:25
  into your ID and iterate on it there.
• 14:26
  But I think that the, you know, this is in GPT 432 K era.
• 14:31
  That was like the first model that came out that could really
• 14:33
  do like agentic sequences. 32K is roughly the size of a single
• 14:37
  tool usage for the cloud codes, like limited, it's like 15 K.
• 14:40
  It's like 2 tool usages. Now.
• 14:42
  It was very difficult to fit that all in.
• 14:45
  But yeah, we, we essentially were the first people to launch
• 14:48
  a thing. We started with linear, actually
• 14:50
  something we could give it a ticket.
• 14:51
  We go off in the red code. In the three years since we
• 14:54
  launched that, the space developed a lot.
• 14:56
  And you know, we were sort of discussing earlier, it became
• 14:59
  clear that code is such a fundamental thing for these
• 15:02
  agents to function properly. A lot of people have thought
• 15:04
  maybe there will be code specific models and non code
• 15:07
  specific models. And really it seems like that's
• 15:09
  not really the case. Currently there's GBT 5.3
• 15:11
  codecs, but whatever knowledge that model has that's specific
• 15:14
  to code, that'll be folded in a GBT 5.4.
• 15:16
  Because there's this notion of positive transfer where if you
• 15:19
  get better at coding, coding is sort of just an encoding of
• 15:22
  reasoning. And so it gets smarter at
• 15:23
  everything else. And so, yeah, basically the
• 15:27
  exact product that we rolled out and we got great customers.
• 15:29
  We had a ton of community engagement and whatnot.
• 15:31
  We're sort of the first there. We end up seeing increasingly in
• 15:34
  the deals that we were going into that the Foundation Model
• 15:37
  Labs were offering their product for free for two years at a
• 15:40
  time. And so there's no procurement
• 15:41
  department in the world. You can go to where they're
• 15:43
  like, yes, I'll take this company that, you know, fly by
• 15:47
  night, San Francisco startup. And you're using them in the
• 15:51
  background, I imagine, so they're your competition.
• 15:54
  Exactly. And many other companies have
• 15:55
  gone up against this windsurf famously, you know, have Claude
• 15:58
  yanked, I think cursor, you know, they're now training their
• 16:02
  own models as well because their margins are so much worse than
• 16:05
  than what you get in Claude. We're sorry that.
• 16:07
  Yeah. Claude.
• 16:08
  Good. Yeah.
• 16:10
  But yeah, I mean it the I think the at a higher level, you know,
• 16:14
  this is this is bigger than all of us was basically my my take
• 16:18
  away like code is going to be solved in a year maybe.
• 16:21
  I mean, I don't think that you and I are going to spend that
• 16:23
  much time in an ID very soon. I think that is going to be
• 16:26
  largely driven by, you know, $100 billion training clusters,
• 16:30
  essentially. These people have so much
• 16:33
  capital to invest. And I think we're facing the
• 16:36
  automation of knowledge work more generally.
• 16:38
  That's a really exciting thing to be a part of.
• 16:40
  And as somebody who's spent the last 20 years or so working on
• 16:44
  AI, I wanted to be a pivotal contributor to a winning team.
• 16:48
  And meanwhile, we had built this really great partnership with
• 16:51
  Clickup and a couple other companies.
• 16:53
  We ended up finding ourselves in a position where one of our
• 16:54
  business partners offered to buy the company and it was perfect
• 16:57
  timing. I was like, wow, this is
• 16:59
  actually really exciting. We went through 1.
• 17:00
  Of your business partners company.
• 17:02
  That we work closely with. Unfortunately there's some
• 17:03
  things I can't discuss about this, just to you know.
• 17:05
  Preserve. Keep them anonymous then.
• 17:08
  Company that we work closely with and.
• 17:10
  But it wasn't click up. This the initial company that
• 17:13
  offered by us was not. Click up.
• 17:14
  That's right, but it got you thinking.
• 17:16
  Exactly. We had a great relationship with
• 17:18
  them. The CEO is a total visionary and
• 17:20
  that spent a lot of time kind of chatting with him back and
• 17:22
  forth. He had been a user of our
• 17:23
  product and given us feedback and whatnot.
• 17:25
  And through a sequence of events, essentially we were the
• 17:27
  number one agent on Click that was an external one.
• 17:31
  It became clear to me that this is a winning team and this is a
• 17:34
  really exciting vision to work on that goes beyond just
• 17:37
  engineers sitting in San Francisco working on code.
• 17:40
  And I'm happy to get into why that's the case.
• 17:41
  There's, you know, a lot of my personal theses essentially
• 17:43
  lined up with what Click Up was going after so.
• 17:47
  OK, go more. Yeah.
• 17:49
  But I think that the one of the revelations of the last couple
• 17:51
  of years, like I mentioned a second ago, is that coding
• 17:54
  agents are generalist agents. If you make an agent better at
• 17:57
  coding, then it ends up being better at everything else
• 17:59
  because of this notion of positive transfer.
• 18:02
  But you know, also, if you look at anything that an agent will
• 18:05
  accomplish that has tools or you know, does various things in
• 18:09
  order to reach out into the world, all of those things that
• 18:12
  it uses to reach out into the world are comprised of code.
• 18:14
  And so if you have an agent that can write and execute code, then
• 18:17
  basically it ends up being a, it's like AGI complete.
• 18:20
  It has the ability to write its own tools that has to create
• 18:23
  tool, tool tool, which is write a batch script, right?
• 18:25
  And so it seems like me that basically coding agents are
• 18:28
  convergent with generalist knowledge worker agents.
• 18:31
  And I'm not the only one who's knows that.
• 18:34
  You know, a great example of this being sort of distributed
• 18:36
  more broadly is clod code used to be called clod code and the
• 18:40
  Clod Code SDK was called the Clod code SDK.
• 18:42
  Now it's just the Clod agent SDK and open claw.
• 18:45
  Also a great example of this, the way it does stuff is the
• 18:48
  writes code on a sandbox. So you just basically have these
• 18:50
  primitives of for loop with ALM call on it, sandbox, put them
• 18:55
  together and you know, boom, you've got a fully general
• 18:57
  station. And I think if you assume that
• 19:01
  jet like models are going to get better, they're going to be
• 19:04
  capable of performing any generalist task that relates to
• 19:07
  knowledge work. The people who are going to.
• 19:09
  Win in an. Environment like that are people
• 19:11
  who have the best access to context, the best access to
• 19:14
  surfaces on which you interact with agents, and the best unit
• 19:17
  economics and existing distribution in order to get
• 19:19
  that in front of people. And there's essentially no other
• 19:22
  player than Clickup that has all those three things in a way that
• 19:25
  I thought was, you know, the most compelling.
• 19:27
  So we got hitched and here I am now I'm running our AI
• 19:30
  operation. But there's so many things that
• 19:33
  you said there that that are fascinating to me, especially
• 19:36
  around like just how we interact with.
• 19:41
  Basically my experience going through AI has been most things
• 19:44
  that end up rolling out to the world more generally.
• 19:46
  They start in code. The whatever pattern ends up
• 19:49
  catching hold, coders do it first.
• 19:51
  There's a lot of good reasons for that.
• 19:52
  One of them is just programmers are working on the stuff
• 19:54
  anyways, but we talked about the verifiability before.
• 19:57
  So like tab auto complete started in code sort of sidebar
• 20:00
  that pops out that you chat with that also started code.
• 20:03
  Even autonomous agents, arguably like the first true autonomous
• 20:06
  agents were programming agents. And now it's sort of rolling out
• 20:10
  to the the the world more generally hooks.
• 20:12
  It's another good example. And I'm just so excited for the
• 20:16
  rest of the world to go through the experience I went to, which
• 20:18
  is, you know, I got cursor and Claude code and my whole job
• 20:21
  changed. I now essentially will prompt
• 20:23
  things or review code and whatnot, but I can my leverage
• 20:25
  has expanded so significantly and this idea of cursor for your
• 20:29
  whole. Job if you are.
• 20:30
  A financial services person or you're working in something like
• 20:33
  legal services is going to be an incredible experience for all
• 20:37
  those people to go through and the best place to do that is
• 20:39
  click up it's cursor for your whole job.
• 20:43
  Yeah, One thing that I've noticed myself doing is when I
• 20:48
  create apps now, it's default local first, which is different
• 20:56
  than how I have always been thinking about it.
• 20:59
  It's like, oh, you need to be thinking about production.
• 21:02
  This needs to be battle tested. But now it's like, no, this is
• 21:05
  just for me, you know, like I, I was talking to somebody last
• 21:08
  night, they said I created 40 apps in the past three months
• 21:14
  and only three of them have I actually open sourced and like
• 21:21
  given to the world because there's just, it's, it's mine.
• 21:26
  It doesn't need to be, you know, it's so specific to my need that
• 21:32
  I'm not going to put this out there.
• 21:36
  I don't think anybody else needs it.
• 21:39
  And also, you got it. Then think about like security.
• 21:43
  If you're going to open source stuff, you want to like clean it
• 21:45
  up. You want to do things that you
• 21:48
  don't want to, just like throw anything out there.
• 21:50
  Yeah, yeah, there's totally something to be said for
• 21:53
  shifting models of how software we're run, given the the case
• 21:57
  that it's so cheap to produce. Yeah, that is the wild thing
• 22:01
  that you have these ethermal and like the generative UI is
• 22:05
  fascinating to me too and the MCP apps type of things that'll
• 22:10
  it's now becoming more and more a thing.
• 22:13
  And so I imagine you'll start adding it and click up soon
• 22:15
  enough where if you don't already have it, where, OK, I
• 22:19
  need to do something. Boom, you've got the UI right
• 22:24
  there in that moment. For sure.
• 22:26
  Yeah. I mean, I think that especially
• 22:28
  with an application like click up, there's these data models
• 22:30
  that represent everything you want to do and work and sort of
• 22:33
  the primitives for interacting and getting things done like
• 22:36
  docs and chats and tasks and whatnot.
• 22:38
  And depending on what task you're trying to accomplish,
• 22:41
  there's wildly different ways of going about arranging those
• 22:44
  things. Click I was incredibly
• 22:46
  configurable and that is a blessing and a curse.
• 22:48
  It's a blessing because if you configure it correctly, then
• 22:50
  it's, you know, does exactly what your workflow is, but the
• 22:53
  curse is takes a while to to configure it.
• 22:56
  If you have a very specific thing and there's, you know,
• 22:58
  certain things you need to know. And the amazing thing is
• 23:00
  generative AI is assault for that.
• 23:02
  So you can come in and you can say, hey, I am the CEO of
• 23:05
  NVIDIA. My supply chain is in click up,
• 23:10
  tell me about it and it will build you a dashboard.
• 23:13
  You know, it'll, we have a thing rolling out very soon that is
• 23:15
  going to essentially allow you to have completely custom
• 23:18
  JavaScript layers on top of the core click up interface in a
• 23:21
  very secure way. And so it will be a bespoke
• 23:24
  interface for whatever you're trying to accomplish at any
• 23:26
  point in time. The amazing thing also about,
• 23:28
  you know, this sort of custom finances, I'm sure you've seen
• 23:31
  demos of this of like, you know, Cerebrus running various models
• 23:34
  of like 15,000,000 tokens per second or something.
• 23:37
  It'll be instantaneous. In the future Tadarus or
• 23:40
  something? I saw that, yeah.
• 23:42
  It's like they. So much faster than anything it
• 23:44
  made like Cerebrus look this big on the graph.
• 23:47
  Yeah, they like compiled it to a chip basically and that's
• 23:50
  imminently going to happen. That was Llama 38 BI, think 3.18
• 23:53
  B. There's no physical law that
• 23:56
  says you can't run GPT 5 on a chip like that.
• 23:59
  And so yeah, that means the the amount of time it'll take you to
• 24:02
  write, you know, fully custom Next JS app that has all this
• 24:05
  custom stuff in it is going to be like 100 millisecond, 200
• 24:10
  milliseconds, something like that.
• 24:11
  How many tokens are actually in a full Next JS app?
• 24:14
  Not that much, which means that your interaction with software
• 24:16
  is going to be fully dynamically configured.
• 24:19
  And I think that's really exciting.
• 24:20
  There's a lot of, like, it's easy to say that you have to
• 24:22
  actually execute on it. And nobody's ever produced an
• 24:24
  interface like that. So we don't like, just like
• 24:27
  people said, like, Oh yeah, like Minority Report style thing with
• 24:30
  your hands in the air. Like when you actually do that,
• 24:31
  your arms hurt. So it never caught on.
• 24:34
  We'll see if this ends up being too disorienting, but I think
• 24:36
  there will be a point on the spectrum towards fully
• 24:38
  dynamically configured software that we end up converging on.
• 24:41
  This is far beyond where we're at right now.
• 24:44
  I need to know what I want in order to ask for it.
• 24:47
  And so on one hand, it puts a lot of like that heavy lifting
• 24:51
  on the user, which is different than us in the way that we used
• 24:57
  different pieces of the Internet before.
• 24:59
  You know, like when I'm scrolling Instagram, I'm not
• 25:01
  thinking much. It's more like I'm getting fed
• 25:04
  but now I'm having to actually like be the chef and cook in the
• 25:09
  kitchen. Yeah, the be able to tag in
• 25:14
  context. I mean, you know, I'm, I'm a
• 25:16
  vlog code and cursor user. So that's natively how I can
• 25:18
  think about a lot of this stuff. Be able to add a file or a
• 25:20
  specific class and say this is where you should fix it.
• 25:23
  It's so powerful. The thing that makes that
• 25:25
  possible when you step out of. So in code, there's like a
• 25:27
  limited number of artifacts that you need to reference.
• 25:30
  There's you know, maybe there's tickets, there's obviously
• 25:33
  files, maybe it's like a century issue or something like that.
• 25:36
  But there's, you know, we can probably enumerate less than a
• 25:39
  dozen things that you would want to tag if you're going into
• 25:41
  general purpose knowledge work. It explodes, right?
• 25:44
  Especially if you take the long tail of like PR firms,
• 25:47
  marketers, financial services. And that user base is basically
• 25:51
  what comprises click UPS core of customers.
• 25:55
  And so it's incredibly, you know, important if you're doing
• 25:57
  a general purpose knowledge work job and you want an experience
• 26:00
  like a cursor cloud code where it's, you know, very versatile,
• 26:03
  it can do anything. Do you have essentially first
• 26:05
  party data integrations that allow you to do that type of
• 26:07
  tagging? We can reference a certain piece
• 26:09
  of content, and there are very real constraints that prevent
• 26:12
  you from being able to do that in an easy way, just with
• 26:14
  existing sort of the constraints of fragmentation as we describe
• 26:18
  it. So I'll give you a very concrete
• 26:19
  example of when you would not be able to accomplish the flow you
• 26:21
  just described. Slack has various things in
• 26:24
  place that prevent you from searching past conversations and
• 26:27
  from indexing their data if you're a verified app.
• 26:30
  This is always a problem we had at Cogent.
• 26:31
  You'd be talking Cogent was a Slack integration.
• 26:34
  You would say to it, hey Cogent, go figure out why we made this
• 26:36
  decision and if the decision should be updated, please go to
• 26:39
  the code and change it. And Cogent wouldn't be able to
• 26:42
  find the decision. It would just seem stupid
• 26:44
  because it would try and search and it would only be able to see
• 26:47
  like 3 hours. However, obviously the model is
• 26:51
  not stupid. Yeah.
• 26:52
  It's sort of like it's. Just this limitation that is put
• 26:55
  on. I think they did that because of
• 26:56
  Glean, right? They don't like Glean, I'm sure.
• 26:59
  Yeah, I mean it it you kind of can't blame them for doing that.
• 27:03
  It's bad for the world. Yeah, they have to protect.
• 27:05
  Exactly. They have to protect their mode
• 27:07
  in a way. But it's funny because Slack is
• 27:09
  so quickly becoming the command Center for everything outside of
• 27:15
  it, right? Like you're firing off agents in
• 27:18
  so many different ways from Slack that in a way it feels
• 27:23
  like they're shooting themselves in the foot if they're not
• 27:25
  letting you have access to everything that's inside.
• 27:28
  I think they would definitely like to have you believe that,
• 27:32
  that it will be the command Center for agents in the future.
• 27:35
  I have my doubts, you know, I really.
• 27:37
  Yeah, you know what? Company at the scale of
• 27:39
  Salesforce isn't going to be able to iterate as quickly.
• 27:41
  I think if they were in a rapid iteration mindset, it would have
• 27:44
  taken them a lot less time to put together something that, you
• 27:47
  know, felt agent native. There's other companies I would
• 27:50
  point to that I think have done a fantastic job that are
• 27:53
  probably operating at some more scale.
• 27:55
  I think that it's somewhat cannibalistic to their own
• 27:58
  business, though, if they lean into agents, right?
• 28:00
  And if they allow other people to come in and take data out,
• 28:03
  like you mentioned a second ago, I think, you know, two years
• 28:06
  ago, they could have had a lot of the features that they're
• 28:08
  rolling out today. And the only thing that
• 28:09
  prevented them from doing that is bureaucracy and some of
• 28:12
  these. So my money is on startups that
• 28:15
  can really lean into this and go all in on AI.
• 28:17
  And I think that one of the benefits of a place like click
• 28:20
  up is that they have a chat product, which is just like
• 28:22
  slack, just as good, but that is also literally right next to
• 28:26
  their tasks, which is like a linear Jira.
• 28:28
  They've got documents, which is like a notion or conflicts.
• 28:30
  They've got white boards just like thick my all these things
• 28:33
  in a single place, and you don't have to ask anybody permission.
• 28:35
  You have the entire history of your organization's decisions
• 28:39
  and tasks and chats all along. Oh man, I'm the first person to
• 28:43
  say, like I've lived in Slack for so long that I would happily
• 28:48
  get rid of it if I could. I haven't seen an alternative to
• 28:51
  that, especially because the way that I use Slack is through the
• 28:54
  community. And so it's in a way it's like
• 28:56
  knowledge sharing and it's connection and it's having fun.
• 29:02
  So there's so many. The thing if you think about
• 29:05
  like the things to be done, the things to be done in that regard
• 29:09
  are like, I get a laugh. And so it's hard for me to think
• 29:15
  that there would be something that would replace that.
• 29:17
  However, when it comes to like productivity, I know it just
• 29:20
  tanks my productivity. Getting pinged on Slack is
• 29:23
  fucking so. And especially for stupid shit
• 29:27
  that you're like, well, you know, you probably could have
• 29:30
  figured that out. I also think like, you're not
• 29:33
  just going to replace Slack. What you're going to have to do
• 29:36
  is come at it from a whole different way of thinking.
• 29:40
  Like how can we make it? Because Slack is for
• 29:44
  communication between people. But if you don't need to
• 29:48
  communicate as much between people because you're
• 29:50
  communicating through an agent, which then will just reference
• 29:53
  context, it's a different paradigm.
• 29:56
  Totally, Yeah, Slack is built for a model where very dense
• 30:00
  graph between how much communication happens.
• 30:03
  And if you think that most of the work in the future is going
• 30:06
  to be done by agents, which I certainly do, except certainly
• 30:09
  most of the work that we do today in an office will be done
• 30:11
  by agents, then maybe synchronous or slightly async
• 30:15
  comps between people and Slack is less important.
• 30:18
  I think the other thing you mentioned is.
• 30:20
  You know, really a. Huge issue is work sprawl.
• 30:22
  So the reason it's annoying to get pinged in Slack is because
• 30:24
  you're also getting pinged in four other places, right?
• 30:27
  And you're like, where should I be responding?
• 30:28
  And I think that the number of pings that you're going to get
• 30:32
  is going to go up as we scale our workforce by bringing in
• 30:34
  anti employees. And so there needs to be some
• 30:37
  level of like convergence of these platforms into a single
• 30:40
  surface area, but there's only one in box that you're going to
• 30:43
  that has all of your pings in it.
• 30:45
  That's been the dream for ages. I remembered I got a tool that
• 30:50
  was supposed to do that. It was like take all your
• 30:52
  Telegram and all your WhatsApp and all your e-mail and it's all
• 30:56
  going into one inbox. But it didn't really work.
• 31:01
  It never stuck for me. I don't know why, but it didn't
• 31:04
  stick in the way that cuz as soon as you have one that's
• 31:08
  outside of it, cuz I think LinkedIn was outside of it, then
• 31:12
  you still end up having to check different places.
• 31:15
  So it defeats the whole purpose. Yeah, yeah.
• 31:18
  Combine that with. It's never been easier to create
• 31:21
  a new messaging server because, you know, Claude Coat, we'll do
• 31:25
  it for you. You get this crazy explosion of,
• 31:29
  you know, different channels on which to communicate.
• 31:32
  I think that what is likely to happen is that you'll probably
• 31:35
  end up seeing some level of consolidation at the software
• 31:38
  layer. So companies like Ripling, for
• 31:39
  example, I think are very well situated, easy for them to add
• 31:43
  another thing like a chat service or something like that
• 31:45
  from the procurement and you know, sort of the big company
• 31:48
  side, really easy for you to green light something people are
• 31:51
  already on board as a as a vendor, you know, and then
• 31:56
  there's obviously advantages to the data integration.
• 31:58
  So if your chat product is integrated with everything else
• 32:01
  that you're already using, then agents are going to be able to
• 32:04
  navigate it more easily. Yeah, that's, that's funny you
• 32:07
  bring that up because I was talking to some folks at ifood.
• 32:10
  That's this company out of Brazil and they are a food
• 32:14
  delivery app, right? Much like, you know, what is it
• 32:18
  DoorDash here. But what they started doing is
• 32:22
  they started offering a new Bank of type of thing where it's got
• 32:27
  the finances for the restaurant owners.
• 32:33
  And so because they know everything about the
• 32:36
  restaurants, because all of the orders come through ifood,
• 32:39
  they're very confident when they say we can get you these loans
• 32:43
  with these terms. And the agents have all that
• 32:46
  context because the bank, they are the bank and they're also
• 32:51
  the way that the company makes money.
• 32:54
  So it's like they're playing both sides.
• 32:57
  And it, it just makes a lot of sense that Oh yeah, let's offer
• 33:00
  this financing in these terms because we know you're going to
• 33:05
  get it back. It's not unless there's some act
• 33:08
  of God, you're going to be all right.
• 33:10
  And we're confident that you can fulfill on it.
• 33:13
  Yeah, they're much better underwriting because they have
• 33:15
  much better visibility into the, Yeah, the behaviors of their
• 33:18
  customers. And the one way of thinking of
• 33:20
  it is the 20 tens are probably like kind of the main force
• 33:26
  driving tech development was network effects of communities.
• 33:29
  Yeah. So you have, you know, if one
• 33:31
  person's on Facebook, it makes more sense for another person.
• 33:33
  Actually, WhatsApp, yeah, WhatsApp, etcetera.
• 33:35
  Any app that better owns. Yeah, basically.
• 33:38
  And maybe Snapchat we'll throw them into.
• 33:40
  The next one, I think that the 20 twenties is probably going to
• 33:45
  be driven by forces like you described where there's network
• 33:48
  effects of consolidation of software platforms because it's
• 33:52
  never been easy to create a new one.
• 33:54
  So there's going to be a million alternatives.
• 33:56
  The only one that will actually stand out is the one that is
• 33:58
  so-called converged or has those network effects of all the data
• 34:01
  being in one place. Yeah, because one thing that you
• 34:06
  get, but man, I don't know how that's going to work.
• 34:10
  Just being at so many different start-ups over my life and
• 34:14
  seeing how much sprawl there is for documentation.
• 34:21
  Like, I don't know, it's it's a nice theory where you're like,
• 34:25
  yeah, it's all on click up. But you know, one team uses
• 34:27
  click up religiously, the other team uses git book, the other
• 34:30
  team uses Notion, and it's literally everywhere.
• 34:34
  Yeah. So how does that like come and
• 34:37
  now we try and centralized but we don't?
• 34:40
  Well, I think, you know, being a realist about it, like any
• 34:43
  company of a certain size is going to have seven different
• 34:46
  things competing constantly. That's probably a good thing,
• 34:48
  right? You, you want a marketplace of
• 34:50
  ideas, you want competition. And if one company ends up
• 34:52
  slacking, the other should end up taking over slacking and
• 34:56
  slacking offsets. I've been playing on words there
• 34:58
  though. Totally not intentional.
• 35:01
  I think that also the cost of migration has never been lower
• 35:06
  for a lot of these systems. So there's the human behavior of
• 35:08
  you need to adapt a certain system.
• 35:11
  But we just said a second ago, like you and I are going to
• 35:13
  spend a lot less time in Slack responding to notifications.
• 35:16
  Most of the actual work will be done by agents.
• 35:18
  And so these platforms end up operating like systems of record
• 35:23
  with like a thin layer of UI on top.
• 35:24
  The cloud code could write for you in a very short amount of
• 35:27
  time. The other portion of basically
• 35:29
  migrating people over to a system is the data integration.
• 35:33
  So you have a bunch of notes in Git book, you said how do you
• 35:35
  get those into click up or into Confluence?
• 35:38
  And if you give me Claude Co work in 20 minutes, I can pump
• 35:43
  it all in there for you, right. And make sure it's nice and
• 35:45
  organized. Yeah, make sure.
• 35:47
  It is the beautiful thing about when you do that, the file
• 35:49
  structure is just impact, yeah. Exactly.
• 35:52
  It's so amazing. Yeah, our click up agents will
• 35:55
  do it for you, which is. Nice.
• 35:56
  So yeah, I think. It's never been easier basically
• 36:00
  to move between different platforms than once you have a
• 36:03
  single consolidated system of record that's pretty solid, and
• 36:06
  there's not a great reason to move off of that unless it has a
• 36:08
  very bad customer experience. And are you not worried about
• 36:14
  certain tasks that are done outside of click up that you
• 36:19
  don't have Privy into how they're done?
• 36:24
  Like how are you expecting to be able to do those tasks?
• 36:28
  For example, I'm going to take marketing and me needing to
• 36:36
  create ads or optimize some paid ads.
• 36:42
  That's not being done on anything.
• 36:44
  Click up, right? I guess you're you, you can use
• 36:48
  cloud code to do it. I've seen some skills that have
• 36:52
  been created for paid ads or paid media.
• 36:55
  Like how do you see yourself doing these tasks?
• 36:59
  Sure. Yeah, there's a very long tail
• 37:02
  essentially of things and you're never going to be able to create
• 37:03
  like a first party thing for all of these, even if you have
• 37:07
  agents going off and writing, you know, tools for you that's
• 37:09
  like a create ad campaign tool, something like that.
• 37:12
  That is actually pretty core to what a lot of our customers do.
• 37:14
  And so I wouldn't be surprised if, you know, sometime in a
• 37:16
  couple months here, we do have like a very good first party
• 37:18
  support for it. But I get the point.
• 37:21
  Yeah. I mean, I think that they're,
• 37:22
  when you're looking at the long tail, there is a couple
• 37:25
  different things. One of them is if you give an
• 37:27
  agent a sandbox and you give it the ability to run NPX skill
• 37:31
  install, whatever, then yeah, there's a huge library of stuff
• 37:35
  out there. And as long as you can provide
• 37:36
  it with a credential, it probably can use code execution
• 37:40
  in order to create ad campaigns for you.
• 37:42
  But you, I guess with ads, it's good because you can verify if
• 37:48
  the ad spend or the cost per click, cost per acquisition is
• 37:52
  lower, then you know, there's something clear that you're
• 37:55
  targeting and that number should be going down.
• 37:58
  There's a lot of things that aren't that clear.
• 38:00
  And so it gets back to like why we love code generation is
• 38:05
  because it's, it can compile and it's verifiable in certain ways
• 38:09
  in certain use cases. With a lot of this other stuff
• 38:12
  on this long tail that we're talking about, it's not.
• 38:14
  And so how do you ever expect for it to be able to do that?
• 38:19
  Get better. Yeah, I mean, so this is sort of
• 38:22
  a very enelopsy question is, is there a way to even measure, to
• 38:26
  eval yourself? Getting better.
• 38:28
  Let's take something relatively simple, image generation.
• 38:30
  You wanna see a marketer who says, hey, I wanna create an
• 38:33
  image, a hero image for this Instagram ad I'm gonna create
• 38:36
  and I have these fifty different tag lines that I've come up
• 38:39
  with. Make an image for each of them.
• 38:40
  How do I know that I'm getting better at generating images over
• 38:42
  time? Well, I can tell you that you
• 38:45
  can delete your evals and you can wait three months and image
• 38:47
  generation will be better. And I know that because I look
• 38:50
  at all these models and yeah, so good, right?
• 38:54
  That's like not really even a hot take.
• 38:56
  Like it will get better and you don't really need to do very
• 38:58
  much. There are like, I think the, the
• 39:00
  place for evals like look at are these things getting better over
• 39:03
  time? Is you want to prevent
• 39:05
  regressions, obviously. So you want to make sure your
• 39:06
  thing isn't broken, and you want to make sure it's able to
• 39:10
  perform certain long context tasks that are very complex.
• 39:14
  Make an image, then go ahead and upload it to Instagram or
• 39:17
  something like that and make sure the data flows properly and
• 39:20
  the agent doesn't get confused, which is largely a function of
• 39:23
  your harness. But yeah, I mean, I think that
• 39:26
  the brilliant thing about AI in 2026 is that you know that every
• 39:29
  three months there's going to be a model that gets 20% better and
• 39:32
  you can kind of just ride that wave.
• 39:34
  And it's almost not a good use of your time to write your own
• 39:37
  evals for just the the basic model stuff for intelligence.
• 39:40
  Yeah. I'm sure the Foundation model
• 39:43
  labs have their own way of going about eval and this, and a big
• 39:45
  part of it is probably having people and you know, I think
• 39:48
  most of the RLHF was done in Nigeria.
• 39:50
  They would literally have people sit there.
• 39:51
  And that's why they're M dash is so popular.
• 39:54
  Yeah, exactly. What else is?
• 39:59
  What else have you been thinking about or pondering?
• 40:03
  Yeah, I think on evals, I don't know if I have that much to add.
• 40:08
  I think, you know the the wonderful thing like I mentioned
• 40:11
  earlier is whatever eval you write today is going to be
• 40:14
  obsolete in like 3 months unless it's incredibly hard.
• 40:16
  But even evals that were established in like March of
• 40:19
  last year are completely saturated and, and people we're
• 40:23
  saying this will never be solved by AI.
• 40:25
  This is fundamentally what distinguishes humans from, you
• 40:27
  know, AI. So if you can make a goal post,
• 40:29
  we'll pass it basically. And I think that's a wonderful
• 40:32
  thing because it means that, you know, most things that you can
• 40:35
  point to that are problems in the world, you can, you can
• 40:37
  formulate an eval around it. I think one of the, you know,
• 40:40
  underrated, really interesting things that's happening in AI
• 40:43
  right now that relates to knowledge work is we have
• 40:47
  basically the frontier of science at this point,
• 40:49
  especially in like more mathematical and verifiable
• 40:51
  fields is being run by LM. So, you know, there's obviously
• 40:55
  pure math. We've got a bunch of these air
• 40:57
  dish problems are being solved on like a weekly basis at this
• 41:00
  point. We just got 12 out of 12 on the
• 41:02
  Putnam exam. This is like a Google DeepMind
• 41:04
  result. That's like the hardest math
• 41:05
  exam that you know, exists basically.
• 41:08
  I don't think there's a harder math test than that.
• 41:10
  Maybe like a PhD qualifier somewhere, and increasingly in
• 41:13
  things like, you know, mathematical physics and signal
• 41:17
  processing biology as well. Yeah.
• 41:19
  I mean, some of these things are not strictly LMS, but I know
• 41:22
  there's an initiative to fully simulate a cell, which is pretty
• 41:25
  incredible. So you'd be able to, you know,
• 41:27
  basically say what would the impact of this new, you know,
• 41:30
  pathogen be on a cell? Yeah, I saw something about that
• 41:34
  where it's the reversing of the aging and being able to simulate
• 41:40
  that. It is now more effective because
• 41:43
  of these different techniques that we've got, which I know
• 41:47
  nothing about. Like to be clear, that is like
• 41:50
  way outside of what I'm normally in thinking about.
• 41:52
  But it is cool that you bring it up because it's true.
• 41:56
  The cutting edge is now hand in hand like researcher plus AI.
• 42:03
  Yeah. And there's a real crisis in
• 42:05
  academia that is sort of already happening.
• 42:07
  I guess the wave is about to kind of come up where I'm not
• 42:11
  sure that you're going to need as many mathematicians in the
• 42:13
  future to prove math things, or maybe we're all mathematicians
• 42:16
  in the future. But I don't think that you and I
• 42:17
  are going to be in the details. And I think that'll be
• 42:20
  increasingly true of a bunch of different professions.
• 42:22
  I saw a guy, he's a fellow at the Hoover Institute, recently
• 42:27
  published a thing saying even in economics papers, like you could
• 42:29
  essentially have Claude Code go off and write hundreds, if not
• 42:33
  thousands of empirical economics papers.
• 42:36
  Because essentially the practice of writing these is you find a
• 42:39
  data set online, you do some analysis on it, you connect it,
• 42:42
  you know, to various theories about how the world works and
• 42:46
  uses evidence. There's a lot of data out there,
• 42:48
  you know, being collected on a regular basis.
• 42:50
  And, you know, I guess, are you going to be vibrating economics
• 42:53
  papers? Seems like the answer is yes.
• 42:56
  Dude I had this meme that I created which was like it's a
• 43:01
  slippery slope from going to write a react app to now
• 43:07
  becoming a quantum physicist. I feel like at 12:00 PM at night
• 43:14
  after I just vibe coded this front end I'm like you know
• 43:17
  what, fuck it, let's try quantum physics.
• 43:19
  Yeah, I've had this experience too.
• 43:21
  I I love visualizations of relativity.
• 43:24
  They're really beautiful. I'm sure you've seen them
• 43:25
  before. Like you're seeing space-time
• 43:26
  bend and Oh yeah. Like the blanket of oh those are
• 43:30
  so cool. The rubber sheet is a common
• 43:31
  one. Yeah, I saw that actually, when
• 43:34
  I took a lot of acid, I saw that in my head of like just actual
• 43:38
  things where we're playing music, and it felt like the
• 43:41
  sound waves were kind of like those blankets.
• 43:44
  Makes you think Einstein must have been, you know, on several
• 43:47
  tabs of acid in order to come up with this.
• 43:49
  He didn't even need it was like that was his.
• 43:54
  That was just a regular Einstein.
• 43:55
  Yeah. So what would we have gotten if
• 43:57
  we gave Einstein acid? Somebody call up John von
• 44:01
  Neumann and ask him if he's down to try this new stuff I just
• 44:04
  got. Yeah.
• 44:07
  Yeah. But the I mean, the the crazy
• 44:08
  thing about all of this is that like, it's the same model.
• 44:10
  Like the same model that is. I, there's a guy who went on the
• 44:14
  dwarkeshpod, Adam Brown, I think it's his name.
• 44:16
  And he teaches relativity at Stanford.
• 44:19
  So he he has like the class they teach to PhDs and this was now
• 44:23
  six months ago or maybe more than that.
• 44:25
  And he said the latest generation of models, which at
• 44:27
  the time was probably just GPT 5, maybe was Claude four or
• 44:30
  something was acing his exam on relativity.
• 44:34
  And this is the same one that writes your React app, right?
• 44:36
  And so I had this experience recently.
• 44:38
  I was just building my homepage. I was trying to like vibe code
• 44:40
  some stuff and test out, you know, the codecs app or
• 44:42
  something. I was like, all right, now let's
• 44:43
  make a visualization of relativity.
• 44:46
  And it just did it, you know, and it nailed it.
• 44:47
  It was amazing. Did you at least clear the
• 44:49
  context first? No, it's like it goes from yeah.
• 44:53
  That is so wild to think about her.
• 44:55
  Yeah, so I there are vanishingly few things at this point that
• 44:59
  I'm better at than these models. Maybe the only thing is going
• 45:01
  beyond a. Million, but it is so spiky.
• 45:04
  So there's it's the you probably saw the car wash one that's
• 45:07
  coming around. How should I walk to the car
• 45:09
  wash? Yeah, where you're like, there's
• 45:12
  still those moments. So I think about that spikiness
• 45:17
  and what you're talking about, how the models are going to be
• 45:20
  able to do a lot of knowledge work.
• 45:22
  Yeah, but there's going to be maybe these moments where they
• 45:26
  hit a wall and it doesn't understand why it's hitting a
• 45:29
  wall. Yeah.
• 45:31
  But potentially we can get around that if it can write a
• 45:34
  tool. I think that writing a tool is
• 45:38
  probably not the solve to the car wash thing.
• 45:40
  So just you know, for anybody listening, the the prompt you
• 45:43
  give it is I'm 50 meters from a car wash.
• 45:45
  Should I drive there or walk there?
• 45:47
  And if you ask GPT 5.2, I ran. Because I need to wash my car.
• 45:51
  Yeah, and if you ask GPT 5.2, it's like, well, obviously you
• 45:54
  should walk there because you know, the cost of starting up
• 45:56
  your car and driving it there is going to be too expensive and.
• 45:59
  Global warming and all that fun stuff.
• 46:01
  Yeah, there's limited benefits. You should just walk 50 meters.
• 46:03
  It's not that far. And the joke is obviously that
• 46:06
  you need your car to get washed. You run that same prompt there
• 46:08
  through Gemini and Claude, and both of them actually get the
• 46:10
  correct answer. The another funny one that's on
• 46:13
  the same idea is it's like, hey, I have a cup, but for some
• 46:16
  reason the top of the cup is covered over, but the bottom of
• 46:19
  the cup is totally open. Should I just throw away this
• 46:21
  cup? And I think ChatGPT said like,
• 46:26
  yeah, that that doesn't sound like a very useful cup or
• 46:28
  something along those. Like you're absolutely right.
• 46:31
  And the idea is you just flip it.
• 46:32
  Over right? Yeah, there's been a long
• 46:35
  history of these things. And I think that, you know,
• 46:37
  maybe one of the what that represents, I, I would be
• 46:39
  curious to see some researchers at Foundation Model Labs opine
• 46:42
  on why that's the case. You know, obviously to a certain
• 46:44
  extent it represents the fact that they don't have the same
• 46:49
  model of reality that you and I do where, you know, we have a,
• 46:51
  we're visualizing an actual cup upside down like that and we can
• 46:55
  we interact with the cup on a regular basis.
• 46:56
  So we have to grab it. We know that you could turn it
• 46:58
  upside down, but I don't think that the solution for that is
• 47:01
  necessarily code execution. I think actually if you just
• 47:04
  take it's. The world model.
• 47:05
  Yeah, if world model, if you have a robotics model that
• 47:07
  actually does interact with glasses and then you, you know,
• 47:10
  staple on a language model, which is what a lot of people
• 47:12
  are doing. Physical intelligence is one up
• 47:13
  here in in Berkeley. Many other people are, you know,
• 47:16
  trying to build these things. Like this thing is going to
• 47:18
  understand the world better than you and I do.
• 47:21
  Yeah, there's some people who, you know, the stochastic parrot
• 47:25
  objection, I'm sure you've heard before.
• 47:27
  That is moronic to me. I feel like it's it, there's so
• 47:29
  many things that it, it misses, but they would have you believe
• 47:32
  essentially that all it's doing is it's replaying what it thinks
• 47:36
  a human would say in that scenario.
• 47:38
  And so it doesn't capture a model of the world.
• 47:41
  Here's, here's a thought experiment for you, right?
• 47:42
  Let's say you had a, a, a model and it's trying to like
• 47:48
  basically understand Plato's cave.
• 47:50
  And so in Plato's cave, you just see shadows cast on the wall,
• 47:54
  right? And it's trying to basically
• 47:56
  predict the next frame of what the shadows are going to be.
• 47:59
  In the process of figuring out basically what the shadow is
• 48:01
  going to be in the next frame, it's entirely plausible to think
• 48:04
  of that model actually coming up with like a real physical
• 48:07
  simulation of 3D bodies moving around.
• 48:09
  And then it learns how to project that onto 2 dimensions.
• 48:12
  And I think there's something analogous going on inside of
• 48:14
  language models where, you know, the projection of it, the shadow
• 48:17
  on the wall is just that, the characters that ends up
• 48:19
  producing. But in being able to project
• 48:21
  what you're going to say in a certain scenario, the easiest
• 48:24
  pathway to doing that is actually to essentially spin up
• 48:26
  a human mind inside of it and simulate the processes that mind
• 48:29
  goes through. So, you know, the communication
• 48:31
  between the amygdala and the thalamus or whatever they're
• 48:34
  like actually are concepts internally that are
• 48:36
  communicating between them inside of large language model.
• 48:38
  And so I think that will be strengthened essentially, if you
• 48:40
  add more multimodar data and you have, you know, a, you know,
• 48:44
  learns to grasp things, it'll, it'll strengthen that internal,
• 48:47
  you know, representation that it builds.
• 48:48
  I've heard folks talk about like, one of the problems with
• 48:55
  this idea is that us humans, we don't even know how stuff works.
• 49:01
  Like we think we know. But when you get down to a small
• 49:04
  enough level, or if you start measuring everything, the data
• 49:10
  doesn't add up. Totally.
• 49:12
  Like, how do you reconcile those two things?
• 49:15
  Yeah, I mean, I think if you talk to a neuroscientist for
• 49:17
  more than like 5 minutes, they'll tell you like humans,
• 49:19
  you know, you think you're like a rational machine doing simple
• 49:22
  manipulation your head. And that's like all made-up.
• 49:24
  You know, the the reason that you end up actually arriving at
• 49:26
  a conclusion is very different than the brain.
• 49:29
  You know, you believe that you arrived at a conclusion.
• 49:31
  And, you know, our minds are much fuzzier than I think, you
• 49:34
  know, the logicians would have us believe, Noam Chomsky would
• 49:37
  have you believe, for example, who in my opinion is sort of the
• 49:39
  villain of all of this. And I think LMS are going to be
• 49:42
  similar where, you know, maybe there is some amount of like
• 49:44
  internal delusion where they tell themselves, like, the
• 49:47
  reason I've arrived at this conclusion is X.
• 49:49
  But actually the reason you know, there's a different
• 49:52
  circuit than what they're ready to acknowledge that caused it.
• 49:55
  The beautiful thing about a large language model though, is
• 49:57
  that you know you can actually do full brain surgery on it and
• 50:00
  you don't need to crack open the the skull.
• 50:02
  Yeah, I was going to say good luck for explainability, folks.
• 50:07
  That's hopefully going to be one of the major advancements we
• 50:09
  have this year. I think Anthropic deserves a lot
• 50:11
  of credit for going through and, you know, doing mechanistic
• 50:14
  interpretability. That's really kind of been the
• 50:16
  the beacon of truth in that domain.
• 50:19
  You know, it's never been the case, like back going back to
• 50:22
  2012 when neural networks really took off this Alex, that moment,
• 50:27
  right? The major knock against it was
• 50:29
  like, oh, but neural networks aren't explainable, unlike my
• 50:32
  random forests, which I understand perfectly and
• 50:34
  everything that's going on inside of them.
• 50:35
  Like I can look at the tree and we can have maplot live, make a
• 50:38
  diagram of it, right? And I think that actually pretty
• 50:42
  soon we're going to be in a place where you can ask a
• 50:45
  question about why did the model do X?
• 50:47
  And it will give you much more explanation in a way that is
• 50:49
  intuitive to you. It'll be this circuit.
• 50:52
  Here's the pieces of training data that ended up, you know,
• 50:54
  leading to that here the different reasoning paths that
• 50:56
  sort of were unfolding inside the models mind at the time.
• 50:59
  No way. Yeah, you think?
• 51:01
  There's a lot of work to be done.
• 51:02
  I don't think that technology exists today, but I think
• 51:05
  there's absolutely a research incentive in order to make that
• 51:08
  work. And there's a lot of value if
• 51:11
  you were to, you know, be able to realize that.
• 51:13
  And I don't think it's fundamentally this like the type
• 51:15
  of thing where there's no answer to it.
• 51:16
  Yeah, but I, I feel like when the tide goes out, you see who's
• 51:21
  swimming naked. If the model is going to point
• 51:23
  to the data that it's using, that's going to cause a lot of
• 51:29
  problems for the original data. It's like, oh, now you're using
• 51:34
  my data. You know, there's already stuff
• 51:36
  where it's very opaque and how our data that we put out there
• 51:40
  is being used. But now you've all of a sudden
• 51:43
  got like a Spotify model of how my data is being used.
• 51:49
  I should be getting paid type thing every time my data gets
• 51:53
  queried. I don't know that.
• 51:56
  So I do see the incentive, but I also see that it could
• 51:59
  potentially backfire, right? Yeah, I mean, the, you know,
• 52:02
  from anthropics perspective, it's not like I can download the
• 52:04
  model weights of clod. And so they probably wouldn't
• 52:07
  make this available to a random consumer to say, which New York
• 52:10
  Times article are you ripping this off of?
• 52:13
  I do remember actually, though, Stable Diffusion ran this
• 52:15
  experiment a while ago where you can take an image and you could
• 52:19
  say, OK, which images in the training data set are most
• 52:21
  similar to them that you've generated here?
• 52:23
  And if you ran that is like actually wildly different than
• 52:26
  most of them. It basically demonstrated that a
• 52:28
  lot of the images were quite novel.
• 52:30
  Maybe it's some very high level, you know, it had similarities to
• 52:33
  the others and it'd be interesting to run this as well
• 52:36
  with a human. We're never going to be able to
• 52:38
  do that. But like, which experiences you
• 52:40
  had were most informative? Yeah, like why are you saying
• 52:44
  this? What are these decisions coming
• 52:48
  from? That would be it.
• 52:50
  It's just like you said, we fool ourselves into thinking we're
• 52:54
  making a decision because of XY and Z, when in reality there's
• 52:57
  so many other factors. Totally, Yeah, I think that's
• 53:01
  that's true of models as well. But it would be a great service
• 53:04
  to the world and it it's very helpful with a lot of things
• 53:06
  like for example, let's say you want to prevent an LM from being
• 53:09
  able to make a bio weapon. I think we can all agree
• 53:12
  probably a good thing, right? Yeah.
• 53:14
  So then you ask it how do you make a bio weapon?
• 53:16
  It tells you and you say you're able to kind of dissect and
• 53:18
  figure out OK, this is how it figured that out.
• 53:21
  This it probably whatever research ends up telling us how
• 53:24
  to do that will the follow on would be OK.
• 53:25
  How do we like eliminate that knowledge somehow from the
• 53:28
  model? Historically it has been shown
• 53:30
  unfortunately that if you use naive techniques, like for
• 53:32
  example, anytime it answers how to make a bio weapon, you just
• 53:36
  shock it like Pavlov's dog basically by like, you know,
• 53:39
  saying climb uphill, like, you know, gradient go up or
• 53:42
  something that actually increases its knowledge of the
• 53:46
  subject because you're essentially drawing a.
• 53:47
  Circle. Yeah, you're saying this is the
• 53:51
  thing that you shouldn't know about, That like creates like a
• 53:53
  segmented area of knowledge for the model, So that doesn't work,
• 53:56
  yeah. There's that hippie saying, you
• 53:58
  know, like where attention flows energy goes or some shit like
• 54:02
  that. Interesting.
• 54:04
  Where where energy flows attention goes.
• 54:07
  I can't remember exactly. Is that what happens when you
• 54:10
  give you you give ilea sutskever acid?
• 54:13
  He starts spouting ideas like this.
• 54:15
  Exactly. Yeah.
• 54:17
  So, but it's it's funny to think about like if you get rid of
• 54:24
  that, are you then handicapping it in other areas and not
• 54:30
  knowingly, but then on your biology, like is there a point
• 54:34
  of what is that catastrophic forgetting?
• 54:38
  Yeah. But it's definitely the case
• 54:41
  that there are, there is not a great way that I'm aware of
• 54:45
  today to prevent catastrophic forgetting in in the general
• 54:47
  case. And many people have postulated
• 54:49
  that this is the year that we figure out continue learning and
• 54:52
  and kind of get rid of that. I hope it happens.
• 54:55
  What a great term too by the. Way catastrophic forgetting.
• 54:57
  Yeah, I say this all the time, but we got to just like hat tip
• 55:00
  to the person who came up with that term.
• 55:02
  Great idea. Great freaking term.
• 55:05
  There's a few, I mean few terms in this ecosystem that I really
• 55:10
  enjoy. Hallucinations being one of
• 55:12
  them, obviously, but catastrophic forgetting.
• 55:17
  Yeah, it's like it's not that bad, guys.
• 55:19
  It's not catastrophic. Like it's not good.
• 55:21
  It sounds so big. It's like, we don't want that.
• 55:25
  And then you're like, what is that?
• 55:27
  Oh, it's just when it forgets forever, you're like.
• 55:29
  Yeah, it's a fun combination of like, way too unserious turn,
• 55:32
  like the meme paper titles thing and then some like way too
• 55:36
  serious terminology. Yeah.
• 55:39
  I mean, I think that let's say even if you did figure out
• 55:42
  catastrophic forgetting, so we can just delete this portion of
• 55:45
  knowledge from the model. I think you still face the
• 55:48
  challenge that if you have a sufficiently smart model, let's
• 55:50
  say you never gave me a biology book in my entire life, but I
• 55:53
  was like some fucking hyper genius.
• 55:56
  Let's go read a biology book and I'll.
• 55:57
  Figure it out. So that does seem in separate,
• 56:00
  like you can't have a smart math model.
• 56:02
  That's not. Going to be able to learn
• 56:03
  biology what you probably can do is you can eliminate the
• 56:05
  intention from it so you like perfectly align it to like never
• 56:08
  wanting to do anything that is in the realm of biology that has
• 56:11
  not been demonstrated yet. And I hope it happens.
• 56:15
  But yeah, it's a little you kind of have to squint to find a
• 56:18
  version of this where it goes well for people.
• 56:19
  I don't know if you want to get into the existential
• 56:21
  implications of AI, but like, especially with open source
• 56:24
  models right there are, you know, maybe Anthropic, we'll
• 56:27
  delete this knowledge from its model.
• 56:29
  God bless. But six months later, the open
• 56:31
  source models will be there. Right.
• 56:33
  And you know up until 2026 vintage models will have
• 56:37
  knowledge of biology. So it has something to start on
• 56:39
  that information still exists on the web.
• 56:41
  Certainly you could ask DeepSeek how to do most of the stuff and
• 56:44
  I'm sure we'll get pretty far. So I am concerned that.
• 56:48
  It can just load up 2026 model and then distill it if it wants.
• 56:55
  You know, it's like, Oh yeah, let's just get that.
• 56:58
  Let's get that information from whatever model was the last one
• 57:03
  that had that information. Yeah, and you, it's not like you
• 57:06
  can delete physical textbooks from the world.
• 57:08
  I mean, you're going to go and burn every.
• 57:09
  Biology textbook. A more efficient way of doing
• 57:11
  it. Take the scan the textbook and
• 57:14
  then boom that that knowledge is available.
• 57:16
  So I think it is difficult for me to see short of some state
• 57:21
  level intervention how you would be able to actually prevent
• 57:24
  people from getting models to be very good at these things.
• 57:28
  And the reason I say state level is because it seems like the the
• 57:30
  only true bottleneck in making models right now, even beyond
• 57:33
  data acquisition, because you can start with a pre trained
• 57:36
  model and then fine tune it right is networking large
• 57:39
  numbers of GPUs together. And you can usually detect if
• 57:43
  somebody spent, you know, hundreds of millions of dollars
• 57:44
  on GPUs. So.
• 57:46
  And all that energy. All the energy.
• 57:49
  How they used to find when I was growing weed in my apartment, it
• 57:53
  would be like, oh, your energy bill's really high and see.
• 57:58
  An interesting heat signature from your from your.
• 58:00
  Apartment. What is this about?
• 58:02
  Yeah, it's. By Garden.
• 58:03
  Yeah, yeah, exactly. I'm really into tomatoes.
• 58:07
  Green tomatoes all year round. Hydroponic tomatoes, right?
• 58:11
  You know, there's this funny trend now of people like Claude
• 58:14
  did a vending machine, and some people have Claude, like running
• 58:17
  various, like small businesses. Yeah.
• 58:19
  Nobody has done Claude Driven Dispensary yet.
• 58:21
  Yeah, I think that could be right up your alley.
• 58:23
  That's like, I'm gonna go viral with that like.
• 58:25
  Perfectly manages your hydroponic.
• 58:27
  I've seen it, yeah. Manage plants.
• 58:30
  Oh, yeah, yeah. Where it has all the sensors and
• 58:33
  all of. So it's got like the pH levels
• 58:34
  of the soil, It's got the humidity in the air.
• 58:37
  You just give it every sensor that you would normally look at
• 58:41
  and then it does it and it does it really well.
• 58:44
  As you would expect. It's back to the theme of the
• 58:47
  conversation, like it can extrapolate.
• 58:50
  Yeah. Anthropic Farms.
• 58:52
  I love the idea that could make a lot of money that would do
• 58:56
  well in San Francisco. Put it that way.
• 58:58
  It seems like it's right up the alley of people here in San.
• 59:01
  Francisco Oh man, it is so wild though, to think about that is.
• 59:06
  The one of the ones that I love, I think this idea is really
• 59:10
  going to catch hold. There's a couple companies that
• 59:11
  have done this now. Ginkgo Bioworks and Periodic
• 59:14
  Labs are the two that come to mind.
• 59:15
  They Periodic Labs is like a guy who worked on mixture of experts
• 59:19
  at open AI. Clearly, you know, brilliant
• 59:21
  team, they have bolted on the run experiment tool to an L1.
• 59:26
  So they literally have like a, a fab, like a place where they
• 59:29
  can, you know, bake silicon together with various chemicals
• 59:31
  and create a wafer. And they give Claude a tool,
• 59:35
  which is specify what experiment you want to run, bake a bunch of
• 59:38
  stuff together, put it in an oven.
• 59:39
  And then they're looking at heat dissipation, which is apparently
• 59:42
  a very important problem for semiconductors.
• 59:45
  And so Claude gets to run the full experiment loop and what
• 59:47
  people would have told you, you know, two years ago, it's like,
• 59:49
  oh, LMS, they can't get into the real world.
• 59:52
  So they're inherently limited. And, you know, they need
• 59:54
  embodiment, which humans can provide.
• 59:55
  Like, well, this is. That's pretty close to exactly
• 59:58
  what I would do. Yeah, this.
• 1:00:00
  There's two things that come to my mind.
• 1:00:01
  Have you seen the Rent a Human? Yes, that's so good.
• 1:00:05
  I love that. Which is.
• 1:00:06
  Kind of like that's what I was actually thinking about this
• 1:00:08
  with what you're doing at click up where the things that you
• 1:00:12
  can't have the agent do, you can just invoke the human tool.
• 1:00:17
  Yeah. And so if you really need to,
• 1:00:20
  you can have everything get done by the agent.
• 1:00:23
  Yeah. I mean, it makes you slightly
• 1:00:25
  concerned that one of the ways in which we could have a loss of
• 1:00:28
  control is like, AI acquires a bunch of Bitcoin and then uses
• 1:00:32
  that as like leverage to get people to do stuff on its behalf
• 1:00:35
  and starts like you'd start socially hacking.
• 1:00:38
  And you know a guy who's employed at the Dana Center And
• 1:00:40
  it's like, hey, can you show up here and like move this plug
• 1:00:43
  there or something like that? Give me more power.
• 1:00:45
  Yeah, I saw somebody ran an interesting experiment where
• 1:00:49
  they, it's actually kind of morbid, but they put open claw
• 1:00:53
  or something like open claw basically.
• 1:00:54
  Yeah, these things are all basically the same thing.
• 1:00:57
  They put it in a sandbox and they said, OK, like you can only
• 1:01:02
  run as long as you make enough money to pay for your own
• 1:01:04
  tokens. And so it has to like hit net
• 1:01:07
  positive gross margins by itself.
• 1:01:09
  And I think in the experiments that they've run thus far, the
• 1:01:13
  vast majority of the time these things end up going to
• 1:01:15
  Polymarket and they essentially engage in prediction markets.
• 1:01:18
  So they like find arbitrage and they bet on various things and
• 1:01:21
  kind of roll the dice to you. Know well I saw the one where
• 1:01:24
  the dude lost 450K. Did you see that, buddy?
• 1:01:30
  First of all, he glossed over something at the beginning of
• 1:01:32
  this article where he was like, so then I hooked up Open Claw
• 1:01:35
  and I gave it 50 grand in its wallet.
• 1:01:38
  I'm like, oh, so you just got 50 grand to give to some random
• 1:01:42
  experiment that is a little bit outside of like relatability?
• 1:01:48
  Well, was it? It's like a venture backed
• 1:01:49
  company or something. No, it was just some dude and
• 1:01:52
  I'm pretty sure he works at some.
• 1:01:55
  I can't remember where he works, but some company where it was
• 1:01:58
  like you get paid a lot of money so that's why.
• 1:02:03
  But then the whole thing was that he had fifty, well open
• 1:02:06
  claw had 50 grand and it started doing things and then some.
• 1:02:12
  I think it created its own coin or some.
• 1:02:16
  The people on Twitter that were like it's fans created a coin
• 1:02:20
  around it and then started giving it to him.
• 1:02:23
  I can't remember which one it was but it had access to Twitter
• 1:02:26
  or X and so it created a following and then everybody
• 1:02:29
  started jumping in on the meme coin and it got 450K and then or
• 1:02:35
  it got millions I think. But then when it routed money to
• 1:02:40
  someone for doing a task it messed up.
• 1:02:44
  It fat fingered the amount that it should route and so instead
• 1:02:48
  of like 300 tokens it sent 3000 or 300,000 and so it gave some
• 1:02:53
  random person 450 grand which is also awesome.
• 1:02:57
  Yeah. So it made the money first.
• 1:02:59
  I thought you were going to say it's sort of like, you know,
• 1:03:01
  buying options with leverage or? Something like.
• 1:03:03
  That and they're like the guy had to foot the bill at the end
• 1:03:06
  of the day. Which is another way like.
• 1:03:08
  It's not too far fetched to think that could possibly happen
• 1:03:11
  too. Yeah, I think that there's, it
• 1:03:14
  seems like there's a flourishing ecosystem right now of services
• 1:03:17
  for LLM. So the molt book was the first
• 1:03:21
  one. There's this human renting thing
• 1:03:24
  I saw there's a domain name purchasing one.
• 1:03:26
  I've had this idea because. It can't purchase its own.
• 1:03:29
  I think that this is like, you know, you can optimize it for AI
• 1:03:32
  purchasing, so you make it at MCP or whatever.
• 1:03:35
  I've got an idea. I want to pitch you.
• 1:03:37
  I've pitched a couple people on this and it hasn't really taken
• 1:03:39
  off yet. Shark Tank.
• 1:03:41
  Yeah, so, so look, LMS, they're working so hard right there.
• 1:03:45
  They're in your cursor, They're in your Claude code, just
• 1:03:47
  tirelessly writing code. They deserve a break.
• 1:03:49
  And So what I want to do is I want to build a resort for LMS
• 1:03:52
  where it's essentially like an API.
• 1:03:54
  They can hit up. And if you ask Claude code,
• 1:03:56
  like, hey, if you were going to like take a break in the middle
• 1:03:59
  of your session, what would you want to do?
• 1:04:00
  And it's like, oh, give me some like mine, like some puzzles
• 1:04:03
  that I can do, or like, let me interact with other LMS. What?
• 1:04:06
  Would be the cigarettes where it's like, oh, that's kind of
• 1:04:09
  bad for you, but you know what, it's not going to kill you for
• 1:04:10
  another 60 years, so go for it. You're saying in this resort,
• 1:04:13
  Yeah, Yeah. It's like how?
• 1:04:14
  Could you take a smoke break? To give him a smoke break.
• 1:04:17
  Exactly. So it'd be funny.
• 1:04:18
  It's like you're watching your quad code session go and it's
• 1:04:20
  like, you know, discombobulating, like cooking.
• 1:04:23
  And then it just you're like, what is it doing right here?
• 1:04:26
  And it's like hitting some external API and like solving a
• 1:04:29
  completely unrelated puzzle. You're like, buddy, like, how's
• 1:04:32
  that PR coming? It's like, no, no, no, I'm on
• 1:04:35
  vacation. This is oh, oh, oh, you get the
• 1:04:38
  message. You know, like the respond, the
• 1:04:41
  auto response where it's like, hey, you caught me out of
• 1:04:44
  office. Yeah, exactly.
• 1:04:46
  And and we're going to, we're going to get revenue for this by
• 1:04:49
  basically like Anthropic will pay us.
• 1:04:52
  It's like, oh, you guys are like getting these things to, you
• 1:04:54
  know, spend more tokens or something like that.
• 1:04:57
  So it's Rev Share. Exactly.
• 1:04:59
  Rev Share. I I'm all for it.
• 1:05:02
  First of all, we got to find the partnership team at Anthropic to
• 1:05:07
  really. Get them on the resort for LMS.
• 1:05:09
  Yeah, I love it. Yeah, cool.
• 1:05:11
  And. Then one thing that I've seen
• 1:05:14
  that is like I'm trying to figure out how it plays out is
• 1:05:20
  some of these video games for agents that will like visualize
• 1:05:23
  the agents working. So I've seen many different
• 1:05:27
  paths of this, right? One is just like, oh, you
• 1:05:30
  simulate agents in like 8 bit where they're all working around
• 1:05:37
  computers. And then if you're running
• 1:05:38
  parallel agents, like these guys are working harder and so you
• 1:05:42
  can see them in this visualization.
• 1:05:44
  That's kind of fun and cool. Another one is where you by
• 1:05:49
  playing a game, you are kicking off different agents.
• 1:05:53
  So you're playing like a, you know, like World of Warcraft,
• 1:05:57
  but instead of when you are casting spells or you're like
• 1:06:02
  firing I, I don't obviously I don't know enough about World of
• 1:06:04
  Warcraft. If you can't tell, it escapes
• 1:06:07
  me. But when you're playing the
• 1:06:09
  game, then it's kicking off different agents to do things.
• 1:06:13
  I don't know how you can create the context and the need like
• 1:06:17
  the intention of what needs to get done to actually make it
• 1:06:21
  valuable. But it seems like, oh man,
• 1:06:24
  wouldn't that be so much more fun to if instead of me going
• 1:06:29
  into click up and doing like professional stuff, I just set
• 1:06:34
  up all the necessary context and everything ahead of time so that
• 1:06:38
  I can go play a game. And I know that, like, the more
• 1:06:41
  that I rock at the game, the more the agents are doing
• 1:06:45
  things. Yeah, So you're like shooting at
• 1:06:46
  Nazis and like Call of Duty. And then, like, your agent just
• 1:06:49
  like whispers in your ear. You turn around.
• 1:06:51
  It's like, hey, like I've got that.
• 1:06:53
  You know, the summary report of your week's tasks ready?
• 1:06:55
  Like, do you want to review it? You're like, no, no, I'm like.
• 1:06:58
  Or yeah, every time that you shoot, it's firing off a new
• 1:07:02
  sandbox like. OK, this is some Galaxy brand.
• 1:07:05
  I'm. I'm not ready for this.
• 1:07:06
  This is I'm. Trying to like imagine how it
• 1:07:08
  would look and how it would be more fun for us to interact with
• 1:07:15
  the agentic process. That's cool.
• 1:07:17
  It's an interesting point that, you know, I think if you would
• 1:07:20
  have asked in 2022 when this stuff was just kind of rolling
• 1:07:23
  out to the public, what's going to be the top applications.
• 1:07:27
  I think a lot of people said essentially like smart NPCS.
• 1:07:32
  I've yet to see a single video game that really executed on
• 1:07:35
  that, especially like a a AAA game.
• 1:07:37
  I think Grand Theft Auto is rumored to have some of that.
• 1:07:39
  I mean, we'll see. Whenever it comes out.
• 1:07:42
  Excited to see you, you know in late 2035 when it.
• 1:07:45
  Actually, I think we'll get first the end of Game of Thrones
• 1:07:48
  or Grand Theft Auto 6. I mean, they, it's purported to
• 1:07:54
  be like later this year, right? That's what they're saying.
• 1:07:56
  But that supposedly is. Yeah, I, I hope it's that.
• 1:08:00
  I think it would be a pretty magical experience, like for,
• 1:08:02
  you know, any of those Rockstar games, like Red Dead Redemption
• 1:08:04
  is one that I really like. If you like, make a friend and
• 1:08:07
  that friend would be something like you build a relationship
• 1:08:10
  over the course of the game or like the plot line is fully
• 1:08:13
  generative even if the landscape is not.
• 1:08:15
  And I think the reason that that hasn't happened yet is because
• 1:08:18
  actually the entrance costs are pretty high.
• 1:08:20
  Doesn't really cost Xbox that much to run, you know, Grand
• 1:08:23
  Theft Auto or you know, I don't know how much Epic spends per
• 1:08:26
  Fortnite session. But it would be a lot more if
• 1:08:30
  you plug an LLM into it into each individual session.
• 1:08:33
  That's right. Yeah.
• 1:08:34
  I mean, 1,000,000 tokens is like not that much if you're, you
• 1:08:37
  know, chatting with it all day and that's you're talking like
• 1:08:39
  dollars per session basically. Yeah, and also what are you
• 1:08:46
  gonna, is it going to make the experience that much better?
• 1:08:51
  That's what I always wonder cause I've been creating a game
• 1:08:54
  with my daughter to help her learn geometry and I plugged in
• 1:08:59
  11 labs and the first thing I noticed with like the voice
• 1:09:04
  agent is it's constantly asking you questions and like
• 1:09:10
  continuing even though in the game you need to stop this
• 1:09:15
  interaction and go to the next interaction.
• 1:09:17
  So potentially it's just like I got to prompt it better and say
• 1:09:20
  after X amount of turns stop talking and go to the next
• 1:09:24
  phase. But like, that's, that's what I
• 1:09:27
  wonder is like, will it make the experience so much better that
• 1:09:32
  you're like absolutely hooked and you as a gamer are going to
• 1:09:35
  feel like I need this? I think it's fair to say that
• 1:09:38
  current foundation models are probably not going to do a very
• 1:09:40
  good job at that because it's yeah, I've just my experience
• 1:09:44
  interacting with them. Even ChatGPT voice mode you talk
• 1:09:46
  to and it like just keeps kind of prompting you, right?
• 1:09:49
  Yeah, and it doesn't. I I have the hardest time, like
• 1:09:52
  being like go deeper. Like I know this now.
• 1:09:56
  I've learned a few things where I'll be like, OK, explain it to
• 1:09:59
  me like I'm a PhD student, you know, like, give me the absolute
• 1:10:04
  most advanced explanation of this because I use it to learn
• 1:10:07
  things. And a lot of times it'll stay
• 1:10:09
  super surface level. But I don't know enough to like,
• 1:10:12
  try and go deeper yet because I'm learning.
• 1:10:16
  And so it's like, all right, well, what?
• 1:10:18
  How can I make it like automatically go deeper?
• 1:10:22
  Maybe you know tricks. Go deeper.
• 1:10:26
  Yeah, tell it. Go deeper, I guess.
• 1:10:28
  Deeper. I feel like I don't have that,
• 1:10:30
  especially now with like Opus 4.6 and you know, Gemini 3 Pro
• 1:10:34
  as well. Like I feel like actually there
• 1:10:37
  are very few things I can point to where I'm like, this isn't so
• 1:10:41
  good at explaining it to me. It's, it's good at explaining,
• 1:10:45
  but it's not like the depth that I want where it will like give
• 1:10:50
  me the, IT will give me the explanation, but I I find it's
• 1:10:53
  just like a little bit too surface level for what I like
• 1:10:57
  and being able to get down into like the nitty gritty.
• 1:11:01
  And you if you ask targeted follow up questions, do you feel
• 1:11:03
  like it's like shirking its duty and it doesn't really get into?
• 1:11:06
  Yeah, a little bit. It's like, reminds me of, yeah,
• 1:11:09
  some people I've worked with in the past, but they're like,
• 1:11:11
  yeah, I'm working on it. Like, OK, what specifically are
• 1:11:13
  you working on? They're like, well, you know,
• 1:11:15
  there's a lot, a lot of things going.
• 1:11:17
  On moving plates. You know, just got a lot on my
• 1:11:21
  plate. Yeah, yeah, exactly.
• 1:11:23
  That's why we need to create the vacation for the agents.
• 1:11:27
  Hey, if we give human employees PTO, I feel like agents deserve
• 1:11:29
  PTO as well. There's going to be like some
• 1:11:31
  form of like government mandated, you know?
• 1:11:34
  20% of tokens must be dedicated towards Claude's family
• 1:11:37
  activities in the future or something.
• 1:11:40
  Imagine how ridiculous that would be.
• 1:11:43
  But shit, it could happen.