Latest / Agentic Conversations (formally mlops.community) / How LiveKit Became An AI Company By Accident

Transcript

How LiveKit Became An AI Company By Accident · Agentic Conversations (formally mlops.community)

• 0:00
  We started Live Kit right in the middle of the pandemic.
• 0:03
  We're really fast. We had a lot of like big
• 0:05
  companies that started to use it.
• 0:07
  Product market fit right from the get go.
• 0:08
  But we didn't have a product. I mean, we had an open source
• 0:11
  project. I got pinged by a really large
• 0:14
  cloud provider and they're like, can we buy you?
• 0:17
  Can we license live kit cloud? Or maybe we'll just kill you
• 0:20
  with open source. You're the accidental AI
• 0:29
  company. Wow.
• 0:31
  I didn't think you were going to open up that.
• 0:32
  But yes, yes we are. Jumping right into it.
• 0:36
  Right in Let's go. How did that happen?
• 0:40
  Oh man, that's a crazy story. I've told it once on a podcast,
• 0:43
  but I'll tell it again since you probably have a bigger audience
• 0:47
  than they do. I think you're overestimating
• 0:50
  me. And my mom listens to this
• 0:52
  podcast and that's about it. I can tell you a funny story.
• 0:54
  My mom listens to a lot of podcasts.
• 0:57
  This one not this one probably. She's a huge fan, though, of
• 1:01
  Dylan Patel. I just interviewed him
• 1:04
  yesterday. Did you?
• 1:04
  That's so funny. Yeah.
• 1:05
  It's so crazy. The sad part is that this
• 1:08
  machine right here crapped out 10 minutes in.
• 1:11
  So it wasn't a podcast, it was a conversation.
• 1:14
  Oh, no, yeah, yeah, he's so smart, my mom told me.
• 1:18
  He's so smart. He wasn't necessarily
• 1:21
  demonstrating how smart he was when we were talking, but you
• 1:25
  should put this part in too. Shout out to Dylan.
• 1:28
  I can tell you a funny story about Dylan too.
• 1:29
  He'll be mad though, if I tell you that story.
• 1:31
  On air, off air, off air, off air.
• 1:34
  We'll talk about that. He's awesome though.
• 1:36
  So how do we become an AI company?
• 1:40
  So Live Kit's been on kind of a journey.
• 1:42
  We started Live Kit right in the middle of the pandemic, kind of
• 1:46
  towards the start. 2020 pandemic was a weird time in the world.
• 1:52
  You couldn't leave your house, right?
• 1:54
  And the only way you could connect with other people was
• 1:58
  through the Internet. And the way people did that
• 2:01
  predominantly was using a camera and a microphone on their
• 2:03
  computer, right? It turns out that when you go to
• 2:11
  do that, like transmit data from your camera on your microphone,
• 2:15
  the Internet, as we mostly have known it for the last 30 years,
• 2:20
  wasn't designed for that Http://standsfor.
• 2:28
  You're testing my knowledge, huh?
• 2:29
  No idea. It stands for Hypertext Transfer
• 2:36
  Protocol. Yeah, I never would have got
• 2:37
  that. So it was designed.
• 2:39
  It was designed for you to transfer text HTML which is just
• 2:45
  text over a network. It wasn't called Hyper Voice
• 2:50
  Transfer Protocol, Hyper Video. It needed to be.
• 2:54
  And in during the pandemic, we needed a protocol for
• 2:58
  transferring data over a network that was rich, richer than text,
• 3:03
  audio and video data from your camera, on your microphone.
• 3:07
  And so there is another protocol for doing that called Web RTC,
• 3:11
  but it's just not commonly used. Up until leading right up to the
• 3:14
  pandemic, there were really only three apps at scale that were
• 3:18
  using that protocol, Google Meet, Discord and Zoom.
• 3:23
  And so Zoom uses a kind of a custom version of it.
• 3:26
  So it should be no surprise that when the pandemic hit, those are
• 3:30
  the three apps that everyone was using because they were the only
• 3:33
  three ones that were you. Yeah.
• 3:34
  They were the ones designed for this kind of world we found
• 3:38
  ourselves in. But if you were a developer that
• 3:41
  was trying to build your own application that needed real
• 3:44
  time audio and video, you had to build a ton of infrastructure
• 3:48
  yourself. Think of it as like you had to,
• 3:51
  you didn't have Stripe for this kind of need.
• 3:55
  You had to go build all of this stuff yourself to interface
• 3:57
  directly with the payment gateway.
• 3:59
  In our world here we're talking about with networks, raw kind of
• 4:02
  network protocols. And so I discovered this
• 4:07
  limitation of how much I would have to build myself by actually
• 4:11
  working on a side project during the pandemic.
• 4:13
  I was working on a clubhouse for companies when clubhouse was
• 4:16
  kind of, I know, right? Forgot about that.
• 4:19
  Throwback. Very pandemic.
• 4:22
  And so I was trying to build a clubhouse for companies, and I
• 4:25
  had to build all of this infrastructure myself to
• 4:30
  actually ship the product. And so I'm like, maybe this
• 4:37
  infrastructure that I had to build myself could be valuable
• 4:40
  to other people. So I pinged my, you know, now Co
• 4:44
  founder David. We had done a company before
• 4:47
  that and sold it to medium. And so I pinged him and I said,
• 4:51
  Hey, I want to like, I want to build a side project, but I also
• 4:57
  think it'd be cool if we could like give this infrastructure
• 5:00
  away to other developers so that they can use it.
• 5:03
  And that was what turned into Life Kit was it was an open
• 5:06
  source project that built real time audio video streaming
• 5:11
  infrastructure in a world where you needed to connect with other
• 5:15
  humans over the Internet. And it was very hard to do so or
• 5:18
  build an application that needed to do so.
• 5:20
  And so that's how we started. We, we were really fast.
• 5:25
  We had a lot of like big companies that started to use
• 5:28
  it, you know, Spotify and Adobe and eBay and Oracle and all this
• 5:32
  stuff. So product market fit right from
• 5:33
  the get. Go it was product market fit
• 5:35
  almost from the get go, but we didn't have a product.
• 5:37
  I mean, we had an open source project.
• 5:38
  Yeah, OK, but we didn't have a commercial product.
• 5:40
  And so we decided, hey, like these big companies are using
• 5:43
  US. Let's go talk to them and find
• 5:44
  out why. And what they said was like,
• 5:46
  hey, we love the product you've built.
• 5:48
  We can see your code clearly. You guys are good programmers or
• 5:51
  my Co founder is maybe not me. So much full credit to him, but
• 5:56
  can we have a commercial product?
• 5:58
  Like we don't want to deploy and scale this ourselves.
• 6:00
  We want you to deploy and scale this for us and run a network
• 6:05
  and we'll pay you money. So we said, OK, well, that's a
• 6:07
  path to continue working on this for the long run, you know, so
• 6:12
  why don't we do it? So we raised some money and we
• 6:14
  started working on this live cloud system, which is like this
• 6:17
  global network for streaming audio and video.
• 6:18
  We won't go too much into the details of it because that's not
• 6:21
  really the question you asked me, is it?
• 6:23
  How do we become an AI company? But I promise this part is kind
• 6:26
  of relevant. So we launched the cloud product
• 6:29
  at the end of 2022 and at the end of 2022, ChatGPT, the
• 6:35
  website comes out and I'm like wow, this GPT 3 model is like so
• 6:41
  good or 3.5 is so good. It feels like I'm texting with a
• 6:46
  human. I'm like, what if I took Live
• 6:48
  Kit and I paired it with this website and like I combine the
• 6:54
  two things together. You know, run chat.openai.com in
• 6:57
  a puppeteer session, a headless browser, and then like take the
• 7:01
  speech, convert it into text, pipe it into the headless
• 7:03
  browser into like the division tag, hit the submit button.
• 7:06
  You're so ahead of. Your time like what if I Jerry
• 7:08
  rigged this thing together where you could talk to chachi PT
• 7:12
  instead of text with it voice say I wasn't a space right.
• 7:15
  I think deep Graham had been around, but I put out this demo.
• 7:19
  I tweet it and I'm like, I'm for sure going viral like.
• 7:24
  Felt it in your boat? I'm like, this is her, like
• 7:27
  Samantha from her. I'm definitely going viral,
• 7:31
  except without the voice and and it just goes nowhere.
• 7:36
  Nobody notices. Well, 90 people faved it, so
• 7:39
  maybe 90 people noticed, but definitely didn't go viral.
• 7:41
  So I was pretty disappointed. Fast forward five months later,
• 7:47
  there's a really long, amazing story then I won't tell because
• 7:50
  it'll take a long time. Something happened in that
• 7:55
  morning that was pretty interesting and now I'm like
• 7:57
  teasing it. Maybe you can just cut it out if
• 7:59
  it's too long. Yeah, we got full director's
• 8:03
  cut. I got, I got pinged by a really
• 8:06
  large cloud provider to I think it was August 4th, 2023.
• 8:14
  So 5 months after I put out this demo, I get pinged by a large
• 8:17
  cloud provider and they're like, I go have lunch with them for
• 8:20
  five hours and they're like, can we buy you?
• 8:23
  Can we license live kit cloud and deploy our own network?
• 8:27
  Or maybe we'll just kill you with open source.
• 8:30
  And I'm like, damn, that's usually what happens after seven
• 8:33
  years. It's happening after two.
• 8:34
  I was like, I'm flattered and I'm also terrified.
• 8:39
  And so they say, well, they're like, look, I'm like, well, how
• 8:41
  much would you offer? Just curious, you know, maybe
• 8:45
  I'll maybe I'll sell, who knows what's the number?
• 8:47
  And they're like 20 mil and I'm like.
• 8:51
  Not a bad payout. Not bad 2.
• 8:53
  Year project. But we raised 15.
• 8:55
  Oh OK, sounds like not the best. Not going to be the best, so
• 9:00
  sorry guys. They're like wait, wait, wait,
• 9:02
  wait. They're like, look, if you can
• 9:05
  somehow tie what you're doing, this audio, video streaming
• 9:10
  stuff, if you can tie it to Jen AI.
• 9:15
  You're like, you didn't see my tweet. 200. 200 million.
• 9:19
  If I could tie it to Jen AI and I said, dude, that's
• 9:22
  ridiculously you said that and I can't tie it to Jen AI have No,
• 9:25
  we're a video conferencing live streaming company.
• 9:27
  I didn't think about my demo. So I leave this meeting.
• 9:33
  Co founder and I are talking, David and I are talking.
• 9:35
  We're like terrified this large companies coming after us.
• 9:39
  What do we do? And we're like, OK, I guess we
• 9:41
  just kind of move fast. That's the only advantage we
• 9:42
  have driving home after, you know, during rush hour, I get
• 9:49
  this e-mail that comes in and it says hello from open AI and I, I
• 9:54
  jump on a call with. Them they found you because of
• 9:56
  what you had. Been like we found your blog
• 9:58
  post and the demo you built that you tweeted and we have been
• 10:06
  wanting to build a voice interface to Chachi PT and so
• 10:10
  three weeks ago we signed up with a personal Gmail Ceta know
• 10:16
  is us. This is this is the content
• 10:20
  right here. Yeah, serendipity and and we
• 10:24
  love the platform and now we want to build this thing for
• 10:27
  real and ship it. And so I was like, what?
• 10:34
  OK. Which you still didn't connect
• 10:37
  the 200 million. At that point you were like.
• 10:40
  No, I got to go back to this guy.
• 10:41
  You know what's funny I say now that I did and I'm like, wait, I
• 10:43
  have a Jedi story. I didn't really actually, I
• 10:45
  didn't connect it. I don't know, I think I was too
• 10:48
  like just mind blown. What's also interesting is that
• 10:53
  so we were, you know, we got in a month of building this with
• 10:56
  them and we shipped voice mode in September, the first version
• 11:00
  of it before advanced and all of the GPT 4 O and all that stuff.
• 11:04
  And in that moment was when I realized that this is an AI
• 11:10
  company and the realization and then sound bite exists
• 11:14
  everywhere, in other places too, because I've said it a lot.
• 11:18
  The realization was, oh man, Open AI wants to build AGI.
• 11:25
  AGI in my opinion, is a human like computer.
• 11:29
  It's a synthetic human. If we're successful and get all
• 11:32
  the way there, synthetic human. How are you going to interact
• 11:35
  with a synthetic human? You're going to talk to.
• 11:38
  It you're going to interact the way that humans interact, like
• 11:40
  how we're interacting, and we use eyes, ears and mouths
• 11:45
  predominantly to interact with one another.
• 11:48
  The equivalent sensors, the equivalent sense is for a
• 11:53
  computer. How it gets that information are
• 11:56
  cameras, microphones, speakers. It turns out that live kit,
• 12:02
  which was built for connecting humans to other humans using
• 12:04
  cameras, microphones and speakers, you can swap out one
• 12:07
  of the humans because all of a sudden that computer is now
• 12:10
  human like and you can use the same technology to connect a
• 12:13
  human to a machine. And so if open AI and Anthropic
• 12:19
  and Gemini and you know, the labs are going to build the
• 12:23
  brain, live kid has an opportunity to build the nervous
• 12:27
  system to carry signals to and from that brain wherever they
• 12:32
  originate and with, you know, ultra low latency.
• 12:37
  And so then I was like, wow, this is going to be, this could
• 12:41
  be, it has a good shot of being a core piece of like
• 12:47
  infrastructure that is the backbone of multimodal AI.
• 12:53
  And in the future, you won't have to qualify it as multimodal
• 12:56
  AI. It'll just be AI, right?
• 12:59
  Because that's where this is all going to go.
• 13:00
  Once the model gets smart enough and capable enough, it's all
• 13:04
  going to be multimodal. So that's how we became an AI
• 13:08
  company. I was not expecting that when I
• 13:11
  asked the question. I did not think that it was
• 13:13
  going that deep. But what a story.
• 13:16
  What a way to just live serendipitously and recognize
• 13:21
  that. It's crazy, you.
• 13:23
  Just rode the wave. You know what?
• 13:25
  Justin Khan is a friend and an investor.
• 13:29
  First check actually into live kit.
• 13:31
  And he was the guy that started Justin TV, right?
• 13:34
  Yeah, Justin TV was trained Twitch and we were just talking
• 13:37
  about this the other day where Emmett one of his Co founders
• 13:40
  who is the CEO of Twitch very briefly, maybe the CEO of open.
• 13:45
  Oh my God, I remember that too. But.
• 13:47
  So many dramas. I once ran into Emmett at a
• 13:50
  party NYC in the early days and I had taken a job at 23andMe.
• 13:55
  And like, anyway, that's not really part of the story I ran
• 13:57
  into. I met at AYC party in the early
• 13:59
  days and I was asking him how is how is Justin TV going?
• 14:03
  It wasn't Twitch yet. And he was like, you know, we're
• 14:07
  doing great. We're running these interactive
• 14:09
  ads for horror movies. We're getting paid.
• 14:12
  But then there's this other thing going on on Justin TV,
• 14:14
  which is everyone keeps streaming StarCraft videos.
• 14:16
  So we're thinking about doubling down on people streaming
• 14:19
  StarCraft videos. And, you know, that became
• 14:22
  Twitch. And throughout my whole career
• 14:25
  as an entrepreneur, I've just been like, why the heck can't
• 14:28
  that happen to me? Why can't I get lucky where
• 14:31
  people just start to use something in a way that I never
• 14:33
  predicted, and then I can just ride the wave into success.
• 14:39
  You can't engineer that, right? And so it's kind of surreal,
• 14:43
  honestly, that it ended up happening.
• 14:46
  I because I've been thinking about like, why can't I just
• 14:47
  have that happen? Why can't I get lucky?
• 14:50
  And then it ended up happening. It's it's wild.
• 14:52
  Maybe there's things that now you're like, OK, I've seen this
• 14:56
  five times and it works out in the end, so I'm not going to be
• 14:58
  so stressed about it. I think it's interesting because
• 15:01
  it's like my fifth company and I have a lot of learnings from the
• 15:05
  previous ones, but I think the learning that I have can be kind
• 15:09
  of like distilled down into one important point.
• 15:12
  It not even necessarily that I've seen it play out and I
• 15:15
  missed something more so that the thing that I didn't do with
• 15:19
  my previous companies was I did not think long term.
• 15:24
  Always think long term when you're doing something right.
• 15:28
  What kind of value are creating long term?
• 15:30
  What kind of movement are you attaching yourself to long term?
• 15:35
  In the past, I've kind of optimized for more reactive
• 15:39
  things like how do I compete with this other person or how do
• 15:45
  I position myself around what is like the zeitgeist at this
• 15:48
  moment and build for that, right?
• 15:51
  Or. You guys don't have an MCP
• 15:52
  server? You know what?
• 15:54
  We actually might, but I. Don't you?
• 15:57
  But that's not what you're talking.
• 15:58
  About we might. That's not what I'm talking
• 15:59
  about, you know, and. That's not what stresses you.
• 16:01
  I imagine whether or not you have an MCP server isn't
• 16:04
  something that keeps you up at night these days.
• 16:06
  It's not and and for what it's worth, Shane built the MCP
• 16:08
  server. If we have 1.
• 16:09
  So blame him. It's I had nothing to do with
• 16:12
  it. I wasn't part of the
• 16:13
  conversation. So it's all about him.
• 16:14
  We're going to talk about that later actually, because he's not
• 16:16
  thinking long term. I'm just kidding.
• 16:19
  Hi, Shane. No, but Shane's awesome by the
• 16:23
  way. But yeah, but you know, like
• 16:31
  we've always optimized in my past companies like around some
• 16:35
  kind of like myopic or small or near term thing.
• 16:39
  What we did differently this time was we actually didn't
• 16:42
  start Livekit to be a company, right.
• 16:44
  We started it as an open source project.
• 16:46
  Because you had a pain. Because we had a pain and I
• 16:48
  actually was. This is the truth DZ my Co
• 16:52
  founder and I we thought that we were going to build a Bitcoin
• 16:56
  wallet. DZ sounds like a Bitcoin name.
• 17:00
  And I think he puts like Bitcoin maxi in his like Twitter, his
• 17:03
  Twitter bio. Maybe he does, I don't know.
• 17:05
  But if he doesn't, he is a Bitcoin maxi.
• 17:08
  So don't talk to him about crypto.
• 17:10
  You'll never hear the end of it. But hi, DZ, he's awesome too.
• 17:14
  But no, we want, we were like, you know what?
• 17:16
  We want to work in crypto. Crypto's getting hot again, see,
• 17:20
  not thinking long term, but in the meantime, like we don't
• 17:24
  really have a good crypto idea. So we'll like just try to build
• 17:27
  some value for people. Hey, you know what, like we're
• 17:29
  in this pandemic and people need to like use audio and video a
• 17:31
  lot. Why don't we make that
• 17:33
  infrastructure easier for developers?
• 17:35
  We've been developers. We've benefited from open
• 17:37
  source. Why don't we give it back and
• 17:39
  like build something so that they can go faster?
• 17:43
  And so that was really what it was.
• 17:44
  We're like, you know, there's going to be more video on the
• 17:46
  Internet, not less. The internet's going to get more
• 17:48
  real time, not less real time. And so that's truly like how we
• 17:52
  started. And then when the AI stuff
• 17:55
  happened, we said, OK, well, we have this magical LLM
• 17:59
  technology, right? We now have a probabilistic
• 18:02
  computer that's like kind of like a raw material.
• 18:06
  How are you going to interact with that raw material?
• 18:09
  How do you harness it and like leverage it in an application
• 18:12
  and deliver that value to users and scale it?
• 18:15
  And so it turns out that you need a lot of infrastructure
• 18:19
  around, like lot of deterministic regular old code
• 18:24
  infrastructure around this, like statistical or probabilistic or
• 18:28
  stochastic computer to really leverage it and harness it.
• 18:33
  And so that's kind of been our guiding light is we're going to
• 18:37
  build all of that infrastructure, all of that muck
• 18:40
  so that developers who are building apps that need to use
• 18:44
  this magical technology don't need to.
• 18:45
  Yeah. And.
• 18:46
  You're the rock. Yeah.
• 18:48
  And then everything else. The Foundation.
• 18:49
  For it, yeah. Yeah, yeah.
• 18:50
  And so I think think long term is the big lesson that I've had.
• 18:54
  Is there a place where you are doubling down your bets right
• 18:58
  now? Yeah, for sure.
• 19:03
  You know, we now the way I like even if you go to our website,
• 19:06
  like we pitched the platform a little bit differently because
• 19:08
  we are kind of all in on AI infrastructure.
• 19:11
  It's not that video conferencing and live streaming companies
• 19:13
  don't use us, they still do. But every product is trying to
• 19:18
  figure out how does AI fit into my product and how can it make
• 19:22
  my product better? And so even video conferencing
• 19:25
  and live streaming is going to have AI as part of it.
• 19:28
  So we're really focused on AI infrastructure.
• 19:31
  What I say now is we're kind of building the platform for voice,
• 19:34
  video, and physical AI, right? The other way that I say it is
• 19:42
  you can now give your application an ability it never
• 19:45
  had before. It can see, it can hear, it can
• 19:50
  speak, and in the robotics case it can move.
• 19:55
  And so what's interesting about where we are today, you know, I
• 20:02
  just gave you the pitch or the one liner, but what's
• 20:04
  interesting about where we are is that we actually don't have
• 20:07
  the complete platform yet. We're not a complete product.
• 20:11
  This is on my kind of anti selling our stuff, right?
• 20:14
  Maybe I should be doing something different.
• 20:15
  But the honest truth is we don't have a complete product yet.
• 20:19
  We have the transport network which is how we started pre AI.
• 20:23
  Then with Open AI, we built this agents framework, which allows
• 20:26
  you to orchestrate and build voice agents.
• 20:28
  And you can think of that as like our next JS, right, For
• 20:32
  voice agents. So we have the next JS and we
• 20:35
  have the network that when you build an agent using our next
• 20:38
  JS, it can connect to our network and stream the audio and
• 20:42
  video. That's nice, but in the middle
• 20:45
  you have like some gaps like so when you build your voice agent
• 20:48
  using our framework, how do you test and evaluate that
• 20:50
  framework? How do you know what it's doing
• 20:52
  right or if it's doing the right thing or the intended thing?
• 20:55
  And then once you have a statistical confidence that it's
• 20:58
  doing the right thing, how do you deploy it and scale it and
• 21:02
  load balance, right? Because voice agents are not web
• 21:06
  applications. They are stateful.
• 21:08
  They're always running for as long as the session is running.
• 21:11
  They're constantly processing voice and generating voice.
• 21:14
  So it's not like a connect, send some data, do a database
• 21:18
  operation, disconnect kind of thing.
• 21:20
  It's stateful and always on. And so how do you deploying load
• 21:23
  balance, something like that? It's just a different paradigm.
• 21:26
  Then of course, you have the run part of it.
• 21:28
  Like so you deploy it. Now how do you run it?
• 21:30
  That's our network infrastructure, that's our
• 21:32
  telephony infrastructure. And then on the end, it's like,
• 21:35
  once you're running it, how do you observe it?
• 21:37
  Where's the data dog or the New Relic for voice AI?
• 21:42
  So shout. Out to Tom, that's another great
• 21:45
  member of your. Team, you know, is I want to say
• 21:48
  something bad about Tom and then say he's awesome afterward.
• 21:51
  But yeah, we got to. I have nothing bad to say about
• 21:53
  Tom. Tom's amazing and and so
• 21:57
  building that end to end platform is really what we're
• 22:00
  trying to do right like and it's not because we are dreaming it
• 22:05
  up and it's like this is what everyone needs and if we build
• 22:08
  that they will come you know field of dream style.
• 22:11
  It's because like our community and users are literally asking
• 22:14
  for this stuff every day. They're like, I built this
• 22:16
  agent, I'm running this agent on the network.
• 22:18
  It's working great, but like I'm struggling with deploying it.
• 22:21
  So we're working on a hosting product.
• 22:23
  I'm struggling with testing and evaluating.
• 22:25
  So we're looking at partnering with people who are working in
• 22:27
  the evaluation space, right? But also we're exploring whether
• 22:30
  we should do some stuff ourselves or an evaluation that
• 22:34
  works really well within the life Kit ecosystem.
• 22:36
  And then the observability side, we're like, OK, well, how do we
• 22:40
  monitor this stuff? People want visibility in these
• 22:42
  systems to know what are the user experiences like that might
• 22:46
  that they're having with my application.
• 22:47
  And so we're kind of trying to tackle all of these things and
• 22:51
  build that kind of all in one platform so that you can go from
• 22:56
  zero to shipping and scaling in production and making money and
• 23:00
  building a successful business all with using live get that
• 23:03
  foundation. You just plug in and then you're
• 23:05
  golden. 100%, yeah. I love that vision.
• 23:08
  And so now it's doubling down on the platform ecosystem.
• 23:11
  Yes, 100% you should pitch this. I'm I think it was better.
• 23:16
  You said it like in like 10 seconds and I didn't have that.
• 23:19
  I took like 2 minutes. Well, that's easy to do.
• 23:23
  One thing I do think about voice, which is fascinating, is
• 23:26
  how much of A rich medium it is and how many signals you can
• 23:31
  get. Yeah, yeah, yeah, from voice,
• 23:32
  which is very unlike text. I mean, you're right about all
• 23:36
  those things, right? Thank you.
• 23:38
  So voice is fundamentally higher bandwidth than text, right?
• 23:43
  It carries more data, you know, it's more dense per packet,
• 23:47
  let's say, right, or per byte. And the reason why is because
• 23:55
  it's carrying more information, right?
• 23:57
  Voice data has, it has prosody and intonation and a cadence,
• 24:05
  all kinds of these aspects of sound that humans like
• 24:11
  automatically pick up on and can change the semantics of what
• 24:16
  you're saying, right? The way you say something
• 24:19
  absolutely matters, maybe not as much as what you say, but it
• 24:25
  definitely makes a difference, right, in interpretability or
• 24:28
  how that information is perceived by your brain right on
• 24:33
  the receiving end. And so the other part about this
• 24:40
  is like, so yes, that's kind of just the law of physics of
• 24:42
  voice. And it simultaneously makes it
• 24:49
  an amazing medium for interaction and a very natural
• 24:54
  medium. I mean, this is what we've been
• 24:55
  doing for thousands and thousands of years, right, is
• 24:59
  using voice to communicate ideas to one another.
• 25:03
  But it also makes it incredibly difficult to build a synthetic
• 25:11
  human like experience in code that matches the fidelity of
• 25:18
  like a human like conversation. And so.
• 25:23
  So I think a lot of possibility, which is exciting, but also it
• 25:26
  requires a lot of experimentation to get right.
• 25:29
  And there's a lot of moving pieces in between, as you said,
• 25:31
  like our framework tries to reduce the amount of entropy in
• 25:35
  the system by handling a lot of things for you, like
• 25:38
  interruptions and turn taking and the low latency back and
• 25:42
  forth streaming of the data. But then you also have things
• 25:45
  like transcriptions and the quality of those.
• 25:48
  When you turn that voice into text and you have the brain
• 25:53
  piece of it, the LLM, is it going to understand those
• 25:56
  prompts in the way that you want them to and you're going to?
• 25:58
  Call that tool correctly. Exactly.
• 26:00
  Is it going to call that tool reliably?
• 26:02
  How long does that tool call take?
• 26:04
  Do you generate a response in the middle so that people don't
• 26:06
  think that you're waiting forever?
• 26:08
  It's like they're all these aspects, conversational dynamics
• 26:11
  and reliability aspects of a conversation that if you don't
• 26:14
  get it right, then you fall into this like uncanny valley and
• 26:17
  then people kind of reject it. And I think what also makes it
• 26:21
  even more difficult is the thresholds for when someone
• 26:28
  rejects it or doesn't think it's useful are very use case
• 26:32
  dependent. So if you are building, for
• 26:35
  example, if you're building like patient Indica hospital, we've
• 26:40
  just been talking to a company. I'll give them a shout out.
• 26:43
  We've been talking to a sword health.
• 26:44
  They're a user of live kit. They're awesome, they're
• 26:47
  growing, they're amazing. One thing that we were talking
• 26:49
  to them about was around latency.
• 26:51
  And they're like, you know, we're like, you know, like the
• 26:53
  latency. How do you feel if we got that
• 26:55
  latency even lower? I don't know, it's maybe coming
• 26:58
  in at like a second or something for a full kind of turns or
• 27:01
  maybe around 750 milliseconds somewhere in that range, 750 to
• 27:04
  a minute to a second for, you know, time between AI speaking
• 27:09
  to you, you speaking to it, the turn latency.
• 27:13
  And they said, you know, honestly, for our use case,
• 27:17
  doesn't matter. They're like when people are on
• 27:20
  the phone that are like doing patient and they get a hospital,
• 27:25
  they don't really care if it takes a second or two seconds.
• 27:29
  They just want to get an appointment on the doctor's
• 27:31
  calendar and they want to talk about the issue that they're
• 27:33
  having and like whether it's like super fast, like, look, if
• 27:37
  it takes 5 seconds or 10 seconds, yeah, it's going to be
• 27:40
  a problem. But the distance between like
• 27:43
  yeah, 200 versus 500 versus a second versus 2 seconds, if
• 27:47
  they're like what we have found is sometimes the LLMS take a
• 27:51
  while to respond. Not a live kid problem, I just
• 27:53
  want to say, but sometimes they take a while to respond the LLM.
• 27:57
  And so it does once in a while hit two 2 1/2 seconds.
• 28:00
  They actually don't see any drop off from that but.
• 28:02
  If you think about a human doing the same thing, there's a lot of
• 28:05
  times where I wait on the. Phone for sure.
• 28:08
  Because somebody is writing something down or they're
• 28:11
  pulling something up on a computer.
• 28:13
  So it's not like we're unused to.
• 28:15
  It's not an unnatural behavior. So that is true, but there are
• 28:20
  use cases where there are high sensitivity to latency.
• 28:23
  And I'll give an example and then I'll talk about a
• 28:26
  technique. So for language learning, speak
• 28:30
  is another company, they build immersive language learning.
• 28:33
  So if you want to learn how to speak a different language, you
• 28:35
  can kind of have like a, an experience where you're talking
• 28:38
  to a native speaker of that language and it feels very
• 28:41
  conversational, like you're in their country and you're talking
• 28:43
  to them and they use live kit for this.
• 28:46
  And latency is very sensitive there because like it's supposed
• 28:49
  to feel like a human conversation.
• 28:51
  And so if you're not responding very quickly, it starts to feel
• 28:55
  unnatural, right? And it breaks the experience or
• 28:58
  the believability of the experience.
• 29:00
  That's the uncanny valley, I guess.
• 29:04
  And so for that use case, latency is does matter a lot.
• 29:07
  What's interesting about what you said about how in like a use
• 29:11
  case like customer support or patient intake, the person is
• 29:14
  like looking up information, they're doing something on their
• 29:17
  computer and then they respond after a while.
• 29:20
  What's interesting is yes that is true, but usually you're
• 29:26
  getting some kind of non verbal feedback.
• 29:28
  You're hearing papers in the background.
• 29:30
  You're hearing paper in the background, you're hearing keys
• 29:31
  clacking or they're or they're saying, you know what?
• 29:34
  I'm going to do something. Can I put you on hold for a
• 29:35
  second? Or they're saying, oh, let me
• 29:37
  see, huh. Like.
• 29:39
  And so either non lexical or actual lexical responses that
• 29:43
  are coming back very, very quickly.
• 29:45
  And then you're like, you're primed.
• 29:47
  OK, I'm going to wait. And so if you just hear silence.
• 29:50
  You're go crazy. That's the part that is like,
• 29:53
  wait, this isn't. This isn't.
• 29:54
  Working. Isn't right?
• 29:56
  Is it broken? Let me ask again, so you know,
• 29:59
  where this is becoming a thing that people are starting to
• 30:02
  think about is in reasoning models because Anthropics model
• 30:06
  and some of the other models now they don't have like a separate
• 30:09
  reasoning and then a separate like regular single shot LLM is
• 30:14
  they have thinking tokens and a thinking budget.
• 30:16
  And you can apply like a certain amount of thought to different
• 30:21
  kind of operations or prompts. And you're like, well, if I'm
• 30:25
  going to like apply a high thinking budget, how do I build
• 30:28
  voice AI around a reasoning model that can like dynamically
• 30:32
  think for different periods of time?
• 30:33
  Well, my recommendation is you need to figure out how do you
• 30:36
  respond and what do you respond with immediately before you
• 30:40
  trigger or in parallel with triggering.
• 30:42
  Kind of like a thinking process. It's so funny you mention that
• 30:45
  because I'm when I was talking to Elliot, who runs VOCA, and
• 30:49
  they do something similar where I'm.
• 30:52
  Actually meeting with the VOCA guys tomorrow.
• 30:53
  Really. Yeah, yeah, yeah, I'm hanging
• 30:57
  with them for coffee. He's one of the deepest thinkers
• 30:59
  on conversational design that I have talked to and he's done
• 31:03
  some really fun stuff where he said we put background noises in
• 31:08
  so that it feels more human. You know, you don't want just
• 31:10
  like a voice agent there that you can't hear anything else
• 31:14
  except for the voice. And then he also said that they
• 31:18
  will intentionally make the agent a little bit like Dumber
• 31:22
  quote, UN quote, because the human tends to enunciate more
• 31:27
  and they tend to be a lot more forgiving and say what they want
• 31:32
  in a very articulate way when they think that like, oh, this
• 31:35
  is just some like shitty AI. Oh wow, and human empathy.
• 31:39
  Exactly right. So funny.
• 31:40
  It's like this trick. It's such a nice little hack
• 31:43
  that you have. And what he was saying too was,
• 31:46
  yeah, you have to mention that you're thinking or you're doing
• 31:50
  something. That action is being taken if
• 31:54
  you're making someone. Yes, because otherwise they
• 31:57
  think that it just went offline. Right.
• 32:00
  There has to be that feedback that you get, right?
• 32:03
  To kind of, yeah, prime your mind as to what's happening,
• 32:06
  right. Even if you can't see what's
• 32:08
  happening, like hearing over hearing it over the phone, you
• 32:10
  need some signal. Dude, I think that's perfect way
• 32:13
  to end it. Is there anything else that you
• 32:15
  want to hit on that we didn't talk about?
• 32:17
  No, it's great, man. Yeah.
• 32:19
  Love chatting? This is very cool, I appreciate
• 32:21
  you doing this.