Latest / Convo AI World / Why Voice AI Is More Than Latency: AssemblyAI on Building Production Speech Systems

Transcript

Why Voice AI Is More Than Latency: AssemblyAI on Building Production Speech Systems · Convo AI World

• Luka Chkhetiani (00:00) I would say reliability is the biggest thing right now because a lot of the models are really, really good, right, on the market, but then you see them break easily in certain conditions. I think just focusing on one single metric, generally speaking, it hides a lot of things. Humans can adapt. Now, when you talk to voice agent, it's really annoying if it's just a monotone. So I think as long as it's natural at the end of the day, giving models reasoning time is okay. Like, I would rather know where our models suck than be lied to that our models are state-of-the-art in every dimension. Hermes Frangoudis (00:45) Hi, and welcome to the Convo AI World Podcast, where we interview the founders, researchers, and leaders pushing the voice AI space forward. Today I'm very excited to have Luka from Assembly AI here with me today. Thanks so much for joining us. Luka Chkhetiani (00:56) Of course. Thanks for inviting. Hermes Frangoudis (01:00) So for anyone that doesn't know AssemblyAI, give us a little bit of background about what AssemblyAI does. Luka Chkhetiani (01:08) AssemblyAI is a speech AI company. We offer state-of-the-art models for real-time and asynchronous transcription, as well as a lot of conversational intelligence features, speaker diarization, basically anything that makes, helps you to make sense out of audio data. Hermes Frangoudis (01:24) so before we get too far into the tech, I love to start every episode with a little bit of origin story. So how did you get into voice AI research and how did Assembly kind of get its start? Luka Chkhetiani (01:36) Good question. I was thinking if I should answer that very honestly or just brand it. it's kind of a combination of what popped out at the time to be the most attractive direction in AI when I started, as well as a little bit of practicality of it. So We've started verbal communication around 200,000 to 100,000 years ago, right? Typewriting has been around for what, 150 years. I am very lazy personally when it comes to typing. Everybody is. It's very, very hard, right? Like talking makes so much sense. So I think there's just some inherent biological bias to communicate verbally because it just information density is much higher. and also at that time it was 2016, '17, when AI third or fourth AI boom was starting, like this generation of AI boom. There are a lot of cool AI applications of systems, but very few of them were the ones that would make person that was not into tech feel like, wow, this is amazing. And I think speech was one of those. In earlier text-to-speech, speech to text models, it was very exciting and I hear it so many times. Wow, computers can do this now. So I think it was just a combination of that excitement and practicality as well. Hermes Frangoudis (02:56) So that's really what kind of pulled you into the voice AI space because I remember speaking with you, you kind of get your start in the AI space more in like deep learning in general, right? So it wasn't always voice AI. Yeah. And how did Assembly get its start? So what was the initial focus for Assembly when you joined? Luka Chkhetiani (03:15) so I've heard this story from Dylan, our founder and CEO. I hope I'm quoting this correctly, but at the time when he started the company, to get a speech recognition product, you needed to kind of purchase a CD disc and it gets delivered to you and write it on the PC and then you use it. The main goal is okay, how do we make this really, really accessible for people? So that was initial goal and when I joined it was around seed stage pretty small, very focused team. And that has been the goal from the beginning. How do we create best models and make them really, really accessible for average developer who doesn't need to be like a voice AI specialist? How can we make it easy for anybody to use it? Hermes Frangoudis (04:03) So really lowering that barrier to entry to the technology, that's amazing. Thank you. so at what point did you realize maybe it wasn't just about speech accuracy, but the overall developer experience? Because you're talking about making it more accessible, so it's gotta tie into that developer experience, right? Luka Chkhetiani (04:22) Yeah, exactly. Good and complex question. I would say those things kind of came together in pair because if you have really amazing product but it's inaccessible and it's not scalable in terms of user experience, doesn't matter, right? Or if it's vice versa, you don't have really good product but it's very scalable, you're kind of, it's the same problem at the end of the day. So I think that just coming from first principles that The entire point of building these models and really focusing in this direction has been how could we be really good at both? Like build good products and also make it accessible for people, because both have to be true at the same time. Hermes Frangoudis (05:04) Yeah. And Assembly AI does a great job. Like I was at your recent hackathon and it was so easy to get your speech-to-text up and running, your voice agents up and running. So kudos to the team. Luka Chkhetiani (05:15) Thank you. Coming from you, that's amazing to hear. Hermes Frangoudis (05:19) So looking back a little bit, what was it that maybe you guys misunderstood about the real-time speech market and you had to kind of change up your approach as you moved from like a research product to something that you release for developers? Luka Chkhetiani (05:35) When I look back, it definitely was how to find a fine line between what customers know they need and what they don't know yet they need. And that's not a criticism, it's extremely early stage of this domain, right? So it's expected that there's a learning curve, we're learning as well, everybody's learning. so finding that fine line Earlier on you could hear that the latency is all the things that matter. And I remember we had a model that achieved 150 milliseconds of emission latency. So that's not a request latency, but when you send the audio chunk and when you receive transcription that is aligned to what was said at the time, so that is pretty much at the edge of human perception of time, right? And we got a few feedbacks that actually it feels a little bit slow and we're like, really? How? So digging into it, it turned out well the better way of optimizing latency was actually turn-to-turn latency. And emission latency had a little bit less weight to decision making. that was very early stage of that product, but it's things like that. Like you hear one thing, if you dig into it they're actually a combination of things and they have different weights. So yeah, it's been period of time where both us and customers are learning about how to do this together. Hermes Frangoudis (07:00) Makes sense, yeah. Bleeding edge of technology. No one's ever written the books before and then when you think about the customer experience from it, like you said, they don't really know what they want, they just know what's not working correct. Right. So they're like, I want this, but really to get there, you have to solve these other problems to get that success. Luka Chkhetiani (07:23) Exactly. and that they can't break each other also at the same time. So it's a complex system. Hermes Frangoudis (07:27) Everything now is a complex system with AI, but it's amazingly complex. so speaking of complexity, where do you think are the hardest gains in speech models today? Is it squeezing out a little bit more accuracy or making them more resilient to the messy real world conversation? Luka Chkhetiani (07:47) Definitely more on this second one. I would say reliability is the biggest thing right now. because a lot of the models are really, really good, right, on the market. but then you see them break easily in certain conditions. So I think There's one question that answers that question really well is how do we build systems that are so robust that your expectation of how they will perform in normal use cases, wide range of use cases, is very deterministic. Hermes Frangoudis (08:23) So it becomes a little bit more repeatable and understood that it's like gonna function generally in this way, depending on how it comes in. Luka Chkhetiani (08:31) Yeah, exactly. While we are optimizing for also that kind of global optimal, so to say. Hermes Frangoudis (08:35) I want to go back a little bit. You mentioned that one of the early things that you guys learned from was that it's not always about the latency, but the turn detection as well. So if developers building voice apps are kind of obsessing over latency, from your perspective, does that still hide something else? Or is that still kind of like a valid concern where these things happen under the hood, but at the end of the day it comes down to a certain other element. Luka Chkhetiani (09:08) It definitely hides some things, obviously. But again, it's fine. It's early stage, we'll get there. We are already seeing a lot of customers are becoming really, really amazing experts at how to evaluate these systems. so things will be much different in one year's timeline or so. I think just focusing on one single metric generally speaking, it hides a lot of things because I Hermes Frangoudis (09:32) Because the one metric doesn't cover all the use cases, right? Luka Chkhetiani (09:36) Right. Exactly. It's like optimizing to be a really good public speaker and only optimizing on your tone. There's like so many things. Like if you have perfect tone of voice doesn't matter, you're not gonna be the best speaker in the world, right? Hermes Frangoudis (09:46) Yeah, there's so many more pieces. So obsessing over just latency is the wrong obsession. It's gotta be more balanced. Like is it latency with word error reporting or word error rate, sorry, the WER as it is. Luka Chkhetiani (10:01) Yeah. It's a combination of things. What I see main problem there is I'm not a human brain expert, but I don't think human brain is generally right now really good at quantifying really complex systems without having a lot of small methods to evaluate certain like subsystems of it. we see that in LLMs as well. Right, they're amazing at benchmarks, they perform very differently for specific use cases. So entire point is how do we test these things as rigorously as possible and try to explain really subjective things as objectively, as quantitatively as possible and then summarize them. Right. And in if models excel at many different dimensions, they are usually pretty good when you do A/B testing afterwards and how they perform with customers in the real world. Hermes Frangoudis (10:52) Okay, so it's about the robustness of the model there, just being able to support the different use cases and being flexible in that case. so when you're designing models and infrastructure for voice agents, what do you guys think of between the trade-offs between accuracy, cost, and speed? Luka Chkhetiani (11:13) Exactly. Hermes Frangoudis (11:14) So it's a loaded question, I get it. Luka Chkhetiani (11:15) Yes. Triggering question. Hermes Frangoudis (11:18) Many long nights worrying about Luka Chkhetiani (11:22) Yeah, how to optimize things in middle point. Well if you build extremely large models you cannot really commercialize them. If you build really small models and focus on price, then customers are not gonna buy it. So what's the best way to do it? I think that your standard like Scientific approach to how to build these things really fit nicely when it comes to technology and economics of it, which is well, what is a problem they want to resolve? What is the simplest and most reliable way to do it? Let's do it and then optimize. But obviously thinking about costs from the beginning is important. you cannot translate like 70 billion parameter models behavior to like two, three billion parameter models. behavior really proportionally. So there's a lot of okay, what's gonna happen if we do this? so starting how do we approach it in a simple way is start with some solid solution to a really clear problem and then try to make it as compact and efficient as possible. That usually works I don't remember a time when that did not work to be fair. Hermes Frangoudis (12:28) Okay. So really kind of building out the larger scale model, solving the core problem, and then seeing how you can refine it for each of these other pieces as to get it through the pipe. Luka Chkhetiani (12:39) Exactly. Yeah. Hermes Frangoudis (12:40) As developers started building with Assembly AI, what were some of the first applications that really surprised you? Luka Chkhetiani (12:47) For me personally, it was definitely robotics. Robotics? Yeah. If I went back like 20 years and I told like child version of myself, you're gonna build models that are gonna be in one of the most prestigious robotics companies, humanoid robots, I would be like, No. That's not real. That was so exciting when I first saw it. Is it the most like commercially crazy application in long term? I think so. but it's just very exciting. it's like all the sci-fi things that I watched, you know, when I was a kid or teenager. It's like, whoa, these things can come alive. Hermes Frangoudis (13:27) I couldn't even imagine the feeling of giving a robot a voice, right? It's super cool. are there particular industries where you guys see speech intelligence has quickly moved from like nice to have to being really critical to the business? Luka Chkhetiani (13:43) Definitely customer service. Definitely. Hermes Frangoudis (13:44) Makes sense. Such high volumes, right? And you only have generally a finite team to service. So how do you and even from my experience when you're dealing with developers and they have questions, the majority of them can be solved just by pointing them in the right direction and giving them a little bit of detail, right? Like they might have missed something. And then only that little extra do you need to really hand hold. So Luka Chkhetiani (14:15) There was some. I don't remember, I think it was one of the bank executives in the US that was reporting that when they really figured out was the kind of conversational path with the customers, over 80% of the calls could have been automated really easily because those questions did not require multi-step reasoning and assessment. Which you can do at pretty good level today, to be fair. so yeah, I think that's been the most impactful. but sales as well, like outreach calls. Hermes Frangoudis (14:50) Right. 'Cause those are people just kinda trying to connect. Yeah. Right. And seeing does this qualify as this the right person to be speaking to? And that's the faster you can do that, the better it moves your business across, right? Makes sense. Luka Chkhetiani (15:04) Yeah, exactly. Hermes Frangoudis (15:05) When teams are bringing speech into production for the first time, what do you think what's the mistake most people make and that breaks? The most common thing to break, I guess. Luka Chkhetiani (15:14) from customers' perspective. Hermes Frangoudis (15:18) Yeah, from a developer's perspective. I'm new to this. Setting up a voice agent. What's the one common mistake everyone kind of makes? You guys have made it so simple that it probably doesn't happen as much anymore. Still Luka Chkhetiani (15:28) Well, we're trying. Long way to go, it's going to change a lot of times. But I would say focusing on metrics that are popular in online material, I would say. In a lot of the situations. My friend was building a voice agent and he forgot I was working at a voice AI company. And He was showing me the implementation methods that were focusing on time to first token. And he was like dissatisfied with the combination of the service. And when I explained to him, okay, well there's these multiple moving parts here, they have to work really nicely together and when you actually draw the line how many mistakes you can make, the amount of mistakes you can make is increasing exponentially as long as the general outcome is acceptable. So time to first token could be really, really late, basically, if your end of turn latency as well as accuracy and interruption rates are you know pretty good, right? So it's like kind of trade-offs. So focusing on one specific metric again, that's the most common mistake. Hermes Frangoudis (16:44) Makes sense. Just being too narrowly focused is always generally a mistake 'cause you kinda have blinders on, right? So it still applies. Luka Chkhetiani (16:52) Yeah. Every industry. Yeah. Technically. Hermes Frangoudis (16:56) In terms of the industry and the competitive landscape, whenever your team sees sensational claims around like word error rate and latency and some of these metrics, how do you interpret those internally and how do you think about them? Luka Chkhetiani (17:07) I love this saying. I'm from Eastern Europe, so I've been hearing this my entire life. If something looks too good, it's probably because it's not true. And that came in really handy in the scientific world. Most of the times if some things are really crazy and you're like, wow, that evolution just accelerates in a major way today. It's 99.99% of the time that's the case, right? so default assessment usually when we see some crazy results are you can estimate like how rigorous were evaluations, what organization is coming from, what's their track record, and so on so on. but usually yeah, it's like skepticism is expected. And also if you ask anybody on our team, they will tell you, I hope everybody else is. skeptical about our claims as well. Otherwise how can we move forward as scientists? And how can we build better systems? so yeah, first assessment is being skeptical, really understanding and digesting what's going on, and then assessing, okay, are these things possible? but those kind of scenarios are pretty rare today. I think we've moved past the Hermes Frangoudis (18:20) Yeah. Yeah, early on it was a lot more common probably. Luka Chkhetiani (18:24) Yeah, you would hear like, we achieved like 2% WER on some internal dataset, and then the question would be, which internal data set also how was the normalization done? And is it corpus level normalization of the metric or is it file level normalization? Hermes Frangoudis (18:43) So all these little things that people could do to kind of push that one metric, that comes back to the narrowness. Yeah. Luka Chkhetiani (18:49) Yeah. You understand it really well because you are so deep into this field. Right. But for a person who is not that much expert in the field, it's like, well, maybe this is absolutely correct. Hermes Frangoudis (19:02) It definitely catches the attention. Exactly. You always have to be skeptical of those things that are just so much further than what the norm would be, right? Like if it's within reason you're like, actually maybe they did find a way to really improve. But how does that work? Luka Chkhetiani (19:13) I like that reasoning, yeah. Hermes Frangoudis (19:16) when you zoom out a bit, what do you think is the real moat around voice AI and like what does that look like today? Like what are the things that really differentiate companies, not just being in voice AI, right? Like what is the core moat? Luka Chkhetiani (19:31) The I would answer the value they deliver, but that seems like too artistic a response. So I would say what kind of products they give customers access to. How good are the products? And not as a standalone product, but other companies making creating more value from those products. And how good is the experience? I think when you think about it, well there's pricing and there's latency and there's infrastructure reliability and again five hundred different things that you should take in consideration. But if you ask just a random person in voice AI, they will tell you, yeah, how much gains can I get in terms of customer conversion if I implement this solution and how much it costs me. Hermes Frangoudis (20:18) Very clear distinction. How successful can I be and what's it gonna do to my bottom line? But that's business, right? Like that's clearly what people care about is optimizing for the one while minimizing for the other. Luka Chkhetiani (20:30) Yeah. And I do think that makes sense. I mean humanity has a lot of good bets and obviously I don't wanna go into political side of this obviously, but we should make more scientific advancement bets that do not have economic outcome today. But for most of the industry, yeah, you need to see economic benefits of doing something which makes a lot of sense. Hermes Frangoudis (20:56) Makes sense. talking a little bit about product. you guys have a new model coming out and it's got some really cool improvements from what I've heard. Can you talk a little bit about that? Luka Chkhetiani (21:04) About who leaked that information. yes, so our model is Universal 3 Pro. that's the one that is coming out is a next generation version of that. And okay, what's cool about it and why should anybody care? so it's a speech LM model and what we have been focusing on is how to make models decision making as intelligent as possible. So speech LLM's entire focus was answering that question to start with. obviously it's also cool architecture from researchers' perspective, but that's not enough reason, right? so it did an amazing job right away on voice AI as well as conversational intelligence tasks. But obviously, since the industry is moving forward, there have been a lot of requests and feedback how to improve it. So this model kind of answers four or five major things. which is wild. Like everybody on the team was like, okay, we really shipped this in this timeline, all of it together. one thing that it introduces is this amazing feature that is agent context. So ASR systems are one-sided transcriptions, right? That's cool for conversational intelligence, but I mean it's like Hermes Frangoudis (22:20) It's only hearing one side of the conversation. Luka Chkhetiani (22:21) Exactly. It's like two people having a conversation, one person basically only listening and relaying to other person. It's like that's not really logical in terms of either realisticness of the conversation or how much data you can get from the call. so this agent context is you can provide the transcript of what agent said to customer. We've seen it, like, halve the metrics, like error rate metrics, various metrics on conversational voice AI, voice agent situations. also we have extended the context window and kind of trained the model in a way that utilizes the history and the context better to infer what's happening now and what's gonna happen next. And third thing that I'm really proud of the team for is speaker revision. I didn't want to make that sound dramatic. Hermes Frangoudis (23:17) Okay. No. Luka Chkhetiani (23:21) But what speaker revision is gonna do is real-time diarization has been huge problem, right? Still is. that's because we're trying to do it in real time and as the audio comes. So it's really hard to differentiate speakers in real time without having all of the clusters to look at them. So what speaker revision does is you get the diarization results as the stream continues, but when the conversation is finished, We rerun the clustering one more time and send you back the corrections. And yeah, it was like double-digit improvements on accuracy. Basically it's asynchronous model accuracy at that point for diarization. Hermes Frangoudis (24:02) It's like mind blowing, right? Like it feels like looking retrospect probably something where you're like, yeah, that totally makes sense. But to think of that solution is really impressive. Luka Chkhetiani (24:14) I'm very happy to hear that. Yeah. Hermes Frangoudis (24:15) So kind of breaking down some of these new features. One you said it's hearing both sides of the conversation. How does that work? Are you then feeding it back what comes out of the LLM as part of the history? Luka Chkhetiani (24:29) Yeah, exactly. It goes as one, you have multiple branches, right? And Hermes Frangoudis (24:33) Yeah, it's like cascading flow, right? You have ASR, you have LLM, you have TTS. Luka Chkhetiani (24:37) Exactly. And also when it comes to just ASR part, there are multiple sources of intelligence that are coming in. So you can prompt instruction prompting or passing keywords. You can enable or disable the history how to say history biasing. there's obviously what is being said from the customer and also agent context. So there's all this Yes. And Hermes Frangoudis (25:00) That's a lot. Luka Chkhetiani (25:03) Turns out it's a lot for a human, but model loves it. It's like it can make much more sense of what's going on and make better decisions quantitatively when all those streams of information come in. Hermes Frangoudis (25:14) That's super cool. So it really can improve that accuracy because it has both parts of the conversation, not just, Hey, I'm gonna yell over here and hope it works. Luka Chkhetiani (25:25) Yes, pretty much. Hermes Frangoudis (25:26) and then you increase the context window. So is that because you're feeding it part of the LLM or are you also feeding it previously like what it said? Like how much of the conversation does it hear as it's going? Luka Chkhetiani (25:38) So I don't want to cite incorrect numbers. it's definitely more than three minutes of context because I believe when we're doing research of okay what's the optimal context size where we don't bias the model too much what happened before. around that time we found the optimal amount of history which does not derail the model as it advances in the conversation because if you focus on too much in the past, it's really hard to move on. That also sounded like life advice rather than model behavior, but yeah, and it's adaptive, right? So that changes and adapts as the conversation flows. Hermes Frangoudis (26:24) So it can kinda like you said, not live in the past, but have just enough context that you're not introducing some sort of long term memory that it has to prequery and things like that. Luka Chkhetiani (26:35) Exactly. That's very precise and much shorter explanation of what I said. Hermes Frangoudis (26:40) it's super interesting. And I'm kinda gonna derail from the script here and go along this line with the increased model intelligence. You talked about being able to prompt the ASR. I've seen that in like some of the TTS models, and it makes me wonder are they really just using like an MLLM as a TTS? But How does that work in ASR? Like you give it an overall direction or like an understanding, a system message? Luka Chkhetiani (27:10) It's a speech LM, right? So it basically prompted the same way as LLM. But the point is, this I don't know if this is a technical problem or economics problem or infrastructure problem, probably combination of all of this. It's like ASR speech LMs are a little bit more specialized in ASR direction, which means you don't get all the benefits that you can get from this huge LM system, right? So what it does and what it's focusing on is how can we have a natural language conversation with a human that is trying to use this model so they can bias it and instruct it to perform this task better in a better way. So it's not gonna tell you what's the weather today and do tool calling. I mean in one year, like 18 months, yes, but it's more about asking model how do you want your transcription system to work. Hermes Frangoudis (28:03) Okay. So more about the focus on the actual problem it's solving versus like just a general broad ability. Luka Chkhetiani (28:08) Yes. It wasn't a pleasant trade-off from research perspective because it's very cool to have like generic prompting. It's like tell it anything and you know it'll perform at some level. But I do think that is important trade-off to make because that's what's delivering value in actuality. Hermes Frangoudis (28:27) Makes sense. And ASR is seeing a lot of different deployment styles. Like you can deploy in the cloud. people are now training models or distilling models, I guess you could say, to the point where they can run on device. Where do you think do you think they're all I don't know if it'll all go on device maybe one day when we have like crazy GPUs in the edge. But when you think about Where is the best performance right now? It's probably still in the cloud, right? Or do you think the on device is getting there? Luka Chkhetiani (29:04) It's hard to answer because I can guarantee there's no human on the planet that can accurately predict that. So for me it's just gonna be a guess, right? Maybe I'm wrong, maybe I'm a little bit right, or maybe I'm right. I mean cloud is definitely most interesting for the industry right now because it Hermes Frangoudis (29:19) Mm-hmm. Luka Chkhetiani (29:20) makes sense. I mean there's a lot of certifications that ensure that your data is very safe and people don't want extra headache like managing all that, you know, deployments internally and on device downloading huge models and so on. The privacy laws on downloading things in certain countries. It's a very complex problem. Is on device increasing and on prem deployments? Yeah, definitely. I wouldn't say it's increasing to a level where it's this is gonna happen in one or two years, it's gonna be only on device. It's increasing, obviously. That makes sense. We have extremely powerful computers with us all the time, so why not make use of them better? Hermes Frangoudis (30:03) Makes sense. and you brought in an interesting topic there. It's not just about like your cloud, but people want to do it on prem. And we're also seeing that with our voice agent deployments, people want to reduce that hop latency because it is just one piece of the puzzle. So is that the bigger use case there is reducing the distance between Luka Chkhetiani (30:24) Yes. Hermes Frangoudis (30:25) the ASR point, the LLM and the TTS? Or is there something more to it for on prem? Luka Chkhetiani (30:31) That's a fantastic question. And we introduced a feature to answer that question. So it's auto routing that we call. So you just start a connection and it's gonna be routed to closest server to you. so you don't have a latency of US West to like east. Like it's gonna be routed to closest to you. Right. that has brought a lot of benefits, but we didn't really see major decrease in on-prem Requests from very large enterprises, the mostly elevated issue of smaller to medium businesses. From what I remember, I don't want to be quoted on this exactly. but when it comes to very large enterprise systems, I think it was mostly about data privacy. I mean patient-doctor conversations and so on, which obviously makes a lot of sense. Hermes Frangoudis (31:23) They want to build it into their own HIPAA environments that they've certified and gotten done. Yeah. Could Luka Chkhetiani (31:27) Could that change as we go forward? Probably it will and it will modify some decision making how people do trade-offs, but yeah, there are always different use cases that require different kind of trade-offs. Hermes Frangoudis (31:44) Yeah, and that's also kind of the difference between like streaming and a batch transcription, right? Because you guys have amazing batch transcription models and you can send it up, but the streaming is a little bit different, right? Like it's about the chunking and how do you create that break point? So right now, do you offer VAD built into the voice agent or is that something people can apply their own VAD and use your ASR. Luka Chkhetiani (32:10) Wow. Hermes Frangoudis (32:14) It's a bit of a loaded question, so we can break it down. Luka Chkhetiani (32:15) No, we shared the topics where we would talk about, right? But this was not scripted. this part then is actually a pretty fascinating question because we're releasing a sync product that kinda answers that question exactly, right? So as you said, there's a batch transcription and latencies are different there and you can submit very long files. But then there's a real time systems which require you to have a continuous connection, right, and its infrastructure is more complex and unless you really, really need it, it's really hard to maintain in large scale. Hermes Frangoudis (32:42) Yeah. Right. Yeah, that streaming infra is like you said, not easy to spin up, maintain, and scale. So what are the options? Yeah. Luka Chkhetiani (32:49) Exactly. So we have this real time product where we just connect and you stream the audio and you get results in real time, right? But to that point that you raised, that answers the question of people who are okay to lift this large infrastructural challenge and they also need that type of complexity for their very complex products, but there's also a lot of people, a lot of customers who have everything else figured out, but need really, really fast and premium transcription. So what we thought about is let's create a sync endpoint. We call it sync endpoint, which is you just post an audio up to two minutes and you get results back in less than 300 milliseconds, 200 milliseconds. Because if you have everything else figured out, you have VADs and you have endpointing and you have all the orchestration. You just need ASR. Customers have asked this to us. Why not build a solution to that? So we'll be releasing that at the same time. And the same model is gonna be powering it. Hermes Frangoudis (33:52) that's really cool. Yeah. this will be out alongside those models, so we should encourage everyone go check out AssemblyAI's new models. Luka Chkhetiani (34:00) Thank you. And this was unscripted also. So pretty exciting. Hermes Frangoudis (34:04) Yeah, it was just where the conversation is leading, these are the more, I feel like, more interesting things. I have some notes here around like you know, how are you collecting feedback from voice agents and you know, like what have you learned about the APIs? I think it's a very valid question, but let's talk about the nitty-gritties of voice AI and VAD. Right now we're seeing an explosion of multimodal, you know, voice-in, voice-out models. And I would love to hear your opinion on this because I've asked researchers about this maybe a year ago now. And one of the things that was interesting to think about is like the multimodals lie. They're not always accurate in the sense that like the audio out does not always match the text out. Do you think that's changed with this latest batch of multimodals? Or do you see that as kind of the inherent flaw of these black boxes? Luka Chkhetiani (34:57) I'm trying to be careful because the answer could annoy different individuals. My personal take on that is it's just a problem of time. Could you say that they're gonna be perfect with this current deep learning approach? Probably not. Most probably not. Are they gonna be good enough to be really heavily used? Yes, it's just a matter of time, right? Like TTS models have been hallucinating a lot earlier on as well. They hallucinate now as well, but they became much more reliable and they are much better when they don't hallucinate. So are ASR systems, right? As time goes by they improve and this duplex models, speech-in, speech-out, they're pretty new. And industry is not really using it so much, and a lot of advancements in deep learning are coming from how much industry is utilizing them. if not, then the moment is kind of iterative where it improves, it's not good enough, then somebody else improves it, it's still not good enough. So at some point it becomes good enough. Hermes Frangoudis (36:03) So that the industry can actually use it and then provide enough feedback to really push it. Luka Chkhetiani (36:06) Yeah, there's inflection points in every domain, right? I remember it was 2017, I believe, and as a gift I gave my friend a printed-out generative AI model that I trained on some Georgian art. He was very pleasantly excited about it, but he was also like, okay, this is pretty abstract. Like what the hell's going on in this painting, right? So that was five years ago. Now today you can generate extremely realistic pictures that you can use for marketing purposes. So it just reached inflection point and then became better and better at some point. I think that's applicable for duplex models as well. Hermes Frangoudis (36:44) So they'll have their place, they'll have their inflection point at some point, but it's not quite there just yet. Luka Chkhetiani (36:52) Yes, I do believe it's gonna accelerate though. If you look at things, things are becoming exponentially fast. So if it would take five years earlier on, probably it's gonna take 18 months now. But again, it's impossible to predict those things, so I'm pretty much pulling those numbers from thin air. Hermes Frangoudis (37:12) It's from your observations, it's from your understanding of the industry. it's an estimate, we'll call it. Luka Chkhetiani (37:16) It's subjective opinions based on objective data. Hermes Frangoudis (37:20) Okay, I'll take that. I like the way that sounds a lot better than estimate. so getting back a little bit more towards the earlier part of the conversation around DevEx and real-time streaming, like when you guys were coming out with your APIs for Assembly. What were some of the most critical pieces of the developer experience? Was it getting the developer connected or making sure that they were properly getting you the data? Like what was the focus of the design of the APIs? Luka Chkhetiani (37:53) What's the best way to cause as least amount of annoyance? I don't know if that's a correct English word, but like let's annoy the users as less as possible. Right. and point is that's kinda negative way to put it, positive way would be Hermes Frangoudis (38:14) Yeah. Luka Chkhetiani (38:15) How can we make it extremely easy for technical and non-technical folks to just integrate this solution? And I think that's a very standard and very principled approach to it. When it becomes a little bit more complex is if you're building complex systems to start with, it's really hard to estimate what people who are not experts in the domain would feel when they're using your product. so you kinda always have to work on trade-offs and get very diverse opinions and so on. But that has been focused all the time. What is the best user experience we can have? the simplest for anyone, literally just integrating API in under five minutes. How can we do that? Hermes Frangoudis (39:00) That's a pretty good approach, is like how do you just keep it as simple as possible? Because that forces you to think about like what's the excess? What are the things that as a developer, like you said, it gets annoying. Yeah. It's like I already defined something here that should tell you something you can infer further down the pipe. Why do I have to define it twice? Luka Chkhetiani (39:18) Yes. And let me tell you, these complex systems require so many different configurations that sometimes when I'm looking at how many different things there are to configure models, I'm like I'm not really sure I know of this. So what we did was well if this is hard for us, can you imagine how hard is it for customers? So we kind of distilled them down to kind of Clusters. For example, when it came to latency versus accuracy trade-offs, instead of managing different VAD and like timing as well as like confidence thresholds and so on, we just came up with minimum latency, balanced, maximum accuracy. If you just want it to be that simple, you just choose one of them and we did all the painful work behind it. If you wanna be really, really technical with it, you can go and actually modify all the knobs you want, but I think that kinda portrays the simplistic approach. Hermes Frangoudis (40:23) No, that makes sense because most people when they're first starting out don't know what they want, right? Like we spoke about this a little earlier. They know where they want to get, but they don't know exactly what's gonna get there. So giving them less choices at the beginning will help you get further along the road, right? Yeah. And then once you get there, you're like, actually, now I know these things, how can I go back and like you said, kinda Luka Chkhetiani (40:50) Definitely. Hermes Frangoudis (40:51) Peel it back and adjust the knobs and keep moving. So no, I think that's a great approach where you make as the experts the informed decision on what you think is the best part of the experience to get the user to that outcome. Luka Chkhetiani (41:05) Yep. I think you put it really well and in a really simple way again. and I love that. It's just about how to allow customers to make certain things exist first. I think that's a very popular phrase in US from well in social media. And then you can make it better later. Hermes Frangoudis (41:24) Yeah. Yeah, you have to just get there because if you struggle to get there, you'll never finish. You'll never get to the point where you can make the you know, you don't want to get lost in the tuning too early on. And I think that's important for customer success and developer success in general, right? Like too many options, they have too much opportunity to mess it up. I Luka Chkhetiani (41:46) I agree with you and I'll take your word also because you're expert in Hermes Frangoudis (41:51) Yeah, we see it at Agora too. It's like because when you think about real-time media streaming and real-time infrastructure, these things are extremely complex. They take a lot of effort to spin up. So it's like how do you give the developer enough power to be successful first? Yeah. I like that. Luka Chkhetiani (42:11) That's good way to put it. Hermes Frangoudis (42:15) You talked about the incremental improvements come from collecting the developer feedback. So how does your team collect feedback from developers about building voice agents and how they use your APIs? So is it like we do workshops, we just rely on customer tickets to tell us like what's going wrong? Like how does the customer's voice come into Assembly? Luka Chkhetiani (42:40) That is also a great question because it touches multiple parts of complex system as well as organization. Easy answer would be literally just jumping on meetings with customers. I enjoy that because it's really easy to distill those things into then research and engineering problems. And also if you listen to five customers, they could be saying different things, but one technical solution could be actually affecting all of them and resolving all of their problems. We've seen those kind of things happening. So how we do that is we tried then this came naturally also to integrate research and engineering teams to customer-facing teams as well as to customer relationships. They love it as well. before this, me and our applied AI lead and our research lead were on customer meeting. a large enterprise customer meeting. And I believe it was extremely impactful because instead of having five meetings internally after that meeting, we kinda all left the meeting with all the information we needed. so that has been really organic way of solving it. Hermes Frangoudis (43:52) Okay. Super cool. Luka Chkhetiani (43:53) I think it works. And so don't touch it. Yeah, well Hermes Frangoudis (43:55) Well you're streamlining the connection between the customer's voice and the engineers that can actually solve their problem. You're not waiting for, like you said, five meetings to kind of distill it and then lose some meaning, lose some of that. But it's also interesting you bring up another interesting topic around like you said, the customer doesn't know what they want. Five people can say five different things and you're like, actually, this is all because of this one thing that they don't really understand that's where it's coming from. 'Cause they're only feeling it from one angle. Right. So super cool. Luka Chkhetiani (44:36) I think to that point though, while that is the case for me, if I personally go to doctor, for example, I don't want to be the guy who has to figure out what type of information to give to doctor to diagnose what I want to be diagnosed. I want the doctor to give me the information and set the reference frames, right? Because they are the expert. I think it goes same to the customers. It's like customers are telling us where the problems and now we have to figure out a way how to diagnose them and maybe find economically viable solution that has largest like effectiveness for our organization as well. If we can do one thing that can solve five problems, why not just do that? I think that's very fair and very expected. Yeah. Hermes Frangoudis (45:23) No, it's one of those things that to you feels like it makes a lot of sense and just happened organically, but you know, for a lot of companies that's just not how they operate. So it's really cool to see like that's the approach that your team is taking where it's like putting the experts in the room, letting them hear the problem and having them diagnose versus like, Okay, let me talk to the support guy who then has to like go back and ask ten more questions. It's funny 'cause one of my questions here says, you know, there's a lot of excitement around voice AI agents. Do you think the technology is right for primetime? And maybe that question made sense like A year ago or six months ago? But I feel like now it's already primetime. Like there's voice agents already deployed. So now that they are in the primetime, how does that How does that filter into improving? So the technology's already being used, right? It's already in production. There's people using these. And is it moving the needle for businesses? The like you have customers that are deploying this across voice agents, across transcription services. And what are some of the cool stories you hear that just make you go, you know what, we made a difference? Can you share some? I know it's tough with customers 'cause it's like privacy around it, so take a minute, think if Luka Chkhetiani (46:41) By the way, it is a fantastic question. trying to figure out how to answer it in an informative way without by saying things that it's all right to say, so When it comes to if it moves the needle or not, I mean economy in that direction is increasing dramatically. So that is the best differentiator in that sense. a lot of people have feelings about maybe certain directions are bubble. I'm not an economic expert, but when it comes to how much value does a certain direction is generating, can pretty much assess that by how conservatively is its performance assessed and how fast is it growing. So for voice AI it's pretty simple because most of the implementations that you have, their economic value can be directly translated because it's customer interactions. They are doing something that can be quantified, it's not a probabilistic system, right? So in that sense I do feel it's very reliable to say that the needle is moving and it's accelerating. How long it's gonna continue to accelerate and where it's gonna reach, again, I don't know. I do think there's a very high limit though. Most of the yeah. Hermes Frangoudis (47:55) No, okay. Luka Chkhetiani (47:56) Most of the like success stories that we see usually The biggest feeling is when you see a lot of individuals getting affected with minimum amount of steps between your model and the customer, right? Because we have large enterprise customers who are with extremely high revenue points and implementations are very elegant and they're solving really hard problems. but it's a lot of steps. So hard to quantify for the brain. But for you customer and like B2C products, it's easier to quantify because wow, this product is used by a hundred million people across the world and it's powered by our model. And the speech transcription is one of the core parts of that product existing. That is very cool to see. Hermes Frangoudis (48:41) So the scale at which you're pushing the industry forward. Super cool. what would you say is probably one of the hardest unsolved problems in voice AI today? Luka Chkhetiani (48:53) Yes. Adaptation. I think we will get much better at everything again, but reliability aside, accuracy aside, those kind of things are intertwined. It's more like how do you adapt the conversation as you go? Because if you're talking to human and they get angry, your response is based on like assessment of the situation, you either neutralize, de-escalate situation or escalate situation, prefer de-escalate obviously. Yeah. Humans can adapt. Now when you talk to voice agent, it's really annoying if it's just monotone. Nothing changes. I'm getting angry because I'm having a conversation for five minutes I cannot resolve a problem, but nothing is changing. So adaptive voice agent systems is unsolved problem. I do think it's gonna take a lot of time because I mean we are humans and it's hard for us to be really effective at communication. A lot of problems today are just communication problems, right? So how do you want to distill that into systems? So it's gonna take some time. Hermes Frangoudis (49:57) Yeah. I think that makes sense. It's like if I'm getting upset, I don't want the model to be like, Hey, it sounds like you're not happy. And yeah, that's even more triggering, I guess you could say, to a person, whereas like a human could read the room. They could understand the person is like clearly agitated, and then maybe, like you said, soften, de-escalate, or redirect to not keep pushing this point that's gonna Luka Chkhetiani (50:24) That's perfect way to put it. Hermes Frangoudis (50:27) Yeah. As voice AI continues to expand as a space, there's always going to be a place for specialized ASR models for like batch transcription and stuff. But in the world of real time, do you see Luka Chkhetiani (50:40) Yeah. Hermes Frangoudis (50:41) Things collapsing more to like multimodal, like we talked about, where as those things improve because it's one model, it can kind of handle a little bit more than a cascading pipe. Or do you think the cascading would get better in terms of how the models play together? Luka Chkhetiani (51:00) Whoa. Really complex. Hermes Frangoudis (51:01) This is the one that makes you think the podcast. Yeah. Luka Chkhetiani (51:03) I mean I love these questions. I think they're gonna become more uniform and I have a few hot takes when it comes to this direction generally. Hermes Frangoudis (51:12) Love to hear hot takes. They're always fun. Luka Chkhetiani (51:15) My personal take is even the reason why async transcription services are very in demand right now is that infrastructure worldwide is not ready for real-time systems, right? So as we move forward, more organizations will want to make sense of data in real time. So things will change as all these things improve. Now cascading systems versus just single multimodal systems, my bet would be that multimodal systems are gonna cover more and cascading will become set of different things. So you'll just coordinate different things that serve very small purposes that doesn't make sense to be kinda put into the model directly. But if I had to bet, yeah, for sure, I do think cascading systems are the way to go today. And they will be perhaps for six to nine months from my personal assessment. of technology at least. but in long term, I totally see these systems becoming reliable enough that they can just do a lot of things and cover larger like blast radius, so to say. Hermes Frangoudis (52:27) So maybe you have a multimodal that specializes in one thing but not another, and in order to have that flexibility you'd have some sort of evolution of the cascade, maybe. Yes. Luka Chkhetiani (52:36) Yeah. They just do different things that are more specialized. It's kind of a way to I don't know, abstract software system generally and kind of really in first principles, right? If one model doing a lot of things is economically viable. Makes a lot of sense to do that. intelligence-wise, you can show well you can kind of show more tasks help models learn better. Hermes Frangoudis (52:59) Yes. Luka Chkhetiani (53:00) so to what point does that make sense? Speech in, speech out, I think that will make sense pretty soon. It's a hard problem, but look at the industry, like people are doing amazing things. Hermes Frangoudis (53:11) This touches upon a question I kind of have been having around when you look at a lot of the frontier models, right? Like OpenAI, Claude, they have these things like thinking models. But you can't use a thinking model in real time. Because you'll talk to it and it'll just you know, three minutes later it'll come in. So that distinction around where it's like maybe the multimodal is the orchestrator behind some of these, or maybe the pipelines get to the point where you're using a voice in through assembly that can also produce like affirmations and things that you don't have to go to a separate TTS. So it like blurs the line because The ASR is clearly not a large language model, but it's large enough to keep the conversation going. Luka Chkhetiani (54:03) I do agree with most of that. Hermes Frangoudis (54:04) Please disagree. No, I'd love to hear the opinion, yeah. Luka Chkhetiani (54:11) Again, I love that type of conversation because there are many different approaches to the same problem, right? I do think reasoning is a good skill to have in generally any type of model that could benefit from it. Now in real time, I think the difference there is you cannot have it making you wait similar way as a coding LLM does. You cannot wait for fifteen seconds, right? But a few hundred milliseconds, yes, you can wait for a few hundred milliseconds to get better response. As we're talking right now, I think about what are you saying to me and processing it and then providing you with the answer based on that, right? It feels pretty natural, even though I'm spending some time thinking and you do the same, right? Hermes Frangoudis (54:57) Exactly. Luka Chkhetiani (54:58) So I think as long as it's natural at the end of the day, giving models reasoning time is okay. And that's by the way one of our biggest North Star research threads at Assembly AI. How do we incorporate real-time reasoning into the models? Because I feel like maybe actually that is what's missing from the systems. Like why we're getting turns. how can we get better turn detection and how can we perform better at like drive-thru situations and recognize somebody's last like proper noun basically that is not a common word, right? Why not? Hermes Frangoudis (55:34) Yeah. No, that's interesting because you talked about a you just brought up a real world implementation like a drive-thru. It's not just about a customer service line where you're calling in, the person's very focused on what they're doing. Drive-thru is like messy, right? People go up, they're like, Hey, what do you blah? You know, there's so many other pieces that the model needs to, like you said, reason around and not maybe jump to the answer. Luka Chkhetiani (55:58) Exactly. And also there's a drive-thru is located near the highway. There's that noise. Technology infrastructure wise is not there, so the microphones are not extremely good at localizing voices. There's a lot of things that prompt certain research directions because of where worldwide infrastructure stands right now. That could change later on, like self-driving cars like if you had amazing electronic signs and all the guidelines on the street, you would need to put less advanced technology in self-driving cars. Like you wouldn't need to detect the like you cannot turn left or right signs. It's just the same way of doing these trade-offs. Reasoning seems to be a good way to kind of give a system time to okay what's going on and what do I need to focus on. Hermes Frangoudis (56:52) I think that's a very good answer and very interesting thought is like right now the challenge is not just the technology in the bubble, right? It's not just a voice agent in a lab somewhere under perfect conditions. They're not using these boom microphones for perfect capture. You're talking about something that was put out there to transmit audio from this thing to like the headset inside, right? Like and humans can make up the difference, but an agent needs to collect all that information and understand how does it push out the noise, how does it elevate the real world request. That's I think that's really interesting. Luka Chkhetiani (57:28) Train-test mismatch, so like simply put. Hermes Frangoudis (57:32) Yeah, exactly. And we're optimizing also for the technology available, so the inputs that are available, not just perfect conditions, but real world conditions, right? Luka Chkhetiani (57:43) Yes, and I do think that's a good way to go about it because if these systems provide a lot of value, that branch will have more money to improve their infrastructure later on and further boost the system. So it's kinda placed together really nicely. Is it hard? Yes. Most of really advanced things are, and that's kind of what's fun about it. Hermes Frangoudis (58:06) Thinking about the industry, what do you think is one of the probably like the most overhyped trends? And what do you think is like underestimated when you think about voice AI in terms of maybe problems or solutions or just in general? Like what's things that people like hype up and you're like, that's not really that impressive versus like this other thing that people are Not talking about as much, but it's long term probably gonna be a bigger thing. Luka Chkhetiani (58:34) See this is where it gets awkward because reasoning takes a lot of time. Hermes Frangoudis (58:38) Yeah. No, it's all right. and maybe this is where like avatar models and things like that'll come into play 'cause like looking at you, I can see your brain is working and you're thinking, right? You're thinking through the response versus like when there's no visual cue, the silence is yeah, is different. Luka Chkhetiani (58:54) Maybe I should introduce some kind of interactive way to Hermes Frangoudis (58:57) Oh, great question. Let me think about this. Luka Chkhetiani (59:02) I do think in terms of and underrating something, look, I'll go more in this might be answering to different set of reasoning, even though technically it answers the question you asked. Underrated thing, evaluations. The importance of evals is very underrated, and I think everybody's just shooting themselves in the foot in the industry. And that is not only voice AI, just everywhere else. Hermes Frangoudis (59:26) Yeah. Luka Chkhetiani (59:27) Lot of times what you see is one organization that is evaluating models releasing certain eval metrics and then second one releasing eval metrics and third one and it's like none of them agree. All of them say different models are state-of-the-art by very large margin and which models are the worst very large margin. So where does it leave us? Where does it leave us is We're spending a lot of resources in that direction, but we're competing. I think when it comes to assessment of system, that's where we don't need to compete in any way. Let's just work together to be fair. Like I would rather know where our models suck than be lied to that our models are state-of-the-art in every dimension. Hermes Frangoudis (01:00:13) 'Cause you can't improve what you don't know. Yes. Luka Chkhetiani (01:00:17) And it's not some kind of internal cycle of just we have to be the best. No, it's that's just how life works. That's how you improve the industry, just know where you fail and just improve it iteratively. So I think that's underrated. Are we gonna get better at it? I think so. Hermes Frangoudis (01:00:33) No, I think you're spot on there. Like the evals is definitely something you don't hear enough of in the industry. Like you hear these people releasing like eval reports and it's like once every so often, like, that's interesting, but how are they running the benchmarks? How are they considering these? Where is this like why is everything on this model perfect? Whereas everything on that model is not. Clearly something is not aligned. All right. We've been talking for a while now and I really appreciate your time, Luka, but I know you got a lot of important things to do, so I'm gonna ask you one last question. It's a little bit of a wild card that I ask all my guests. Yes. Luka Chkhetiani (01:01:13) Pleasure, by the way. Love this conversation. Hermes Frangoudis (01:01:17) And it's been eye-opening. I love hearing from the researchers themselves, because you guys are the ones like really pushing the space forward. It's not just like, my company's the best. It's like, no, this is what we're doing to make it great. But if you were starting to research voice AI and speech AI today, do you think you would start in the same place you are now? Or do you think there's another another emerging opportunity that you'd probably jump onto. Luka Chkhetiani (01:01:43) I mean, speech specific? Hermes Frangoudis (01:01:47) Speech or just in general AI. Like is there another if you're not doing speech AI and you were and most of my guests that I ask this question to, they're like, Well, I'd still be in the AI space, but I'd probably be in this like other portion if I didn't do speech. Where do you think you'd probably land? Luka Chkhetiani (01:02:00) I think I'm a little bit proud person to say that I would land anywhere else. Because my internal reasoning is, well, if I wanted to be anywhere else, I would just do that. Why not? I do think speech AI is very interesting space and I think it just a step way towards more intelligent systems. I've always been interested in generally signal processing. I've been interested in brain a lot and communication. So I think it satisfies a lot of the criteria in terms of what I love doing. I'll definitely be here. And when it comes to approach Like would I take different approach? Like if I could go back and fix lot of scientific or product direction or lot of mistakes, yeah, of course, but I think that's kind of part of it. Hermes Frangoudis (01:02:52) The journey. The journey is part of like where you are and how you got here. It's like more important than just the result, right? Luka Chkhetiani (01:02:55) Yes, I would love to have like 100 perfect answers going forward, but that's not gonna be the case. It's fine. I think I'll just end up here anyway. Hermes Frangoudis (01:03:05) Yeah, no, I think that's a very reasonable assumption because of the things that compel you. It's like a calling in this sense, right? Like engineering in general. A lot of us just get lost in it. And it's that love of the solving the problem that's more important than like, what are other problems to solve? No, no, this is the one that interests me. Luka Chkhetiani (01:03:23) Yes. And people just love different things, right? So it's Hermes Frangoudis (01:03:27) it's awesome to hear. Thank you so much for your time, Luka, and thank you to everyone watching. It's been a pleasure, and we'll see you on the next one. Like, subscribe, do the social media thing. Thank you. Thank you for inviting.