Latest / Agentic Conversations (formally mlops.community) / Product Engineering for LLMs // LLMs in Production Conference Part III // Panel 2

Transcript

Product Engineering for LLMs // LLMs in Production Conference Part III // Panel 2 · Agentic Conversations (formally mlops.community)

• 0:07
  Charles, very good to have you here.
• 0:09
  We have a big panel right now and I'm going to start to bring
• 0:12
  on, do you want to like introduce people?
• 0:15
  And then we'll like, dramatically push them onto the
• 0:17
  screen. I love that.
• 0:19
  Yeah, to thunderous applause. Yeah.
• 0:22
  Adam. Yeah.
• 0:24
  So I guess I'll start by introducing our first panellist,
• 0:31
  Shyamala Prayaga, who is a Senior Software Product Manager
• 0:36
  at NVIDIA. Shown some great work on voice
• 0:40
  assistants in. And I'm looking forward to
• 0:44
  hearing from her about how L Ones have changed that
• 0:47
  landscape. Nice.
• 0:49
  Tomorrow, I think you're here. Great.
• 0:53
  Keep on bringing folks on the stage.
• 0:54
  Next up we have Sahar who is currently doing product at
• 1:00
  Stripe, has worked as an engineer and an engineering
• 1:03
  manager, which is exciting. The topic of this panel is
• 1:07
  product engineering for language models.
• 1:09
  So at this intersection of making a useful product, people
• 1:13
  like and make a good computer do stuff when it doesn't want to.
• 1:17
  That we got a lot of folks with that intersection of skills and
• 1:20
  Sahar is one of them. I'm excited to have them all.
• 1:24
  Hey everyone great to be here and I just left to strike a few
• 1:27
  weeks ago but same knowledge I would like to share so.
• 1:30
  Ohh, awesome. You're still wearing the shirt.
• 1:36
  Yeah. All right.
• 1:36
  And then Next up we have Sarah Guo, who is founder of
• 1:42
  Conviction VC purpose built for software 3.0.
• 1:46
  And it's exciting to hear from the capital providers who make
• 1:49
  this possible with their perspective is on how we're
• 1:51
  going to make you know a new kind of software with with good
• 1:54
  design and good engineering. Great to be here.
• 1:58
  And then last and of course not least, we have William PNR who
• 2:03
  created Feast and then we're detect on so some familiar names
• 2:07
  maybe for folks in the ML OPS community.
• 2:09
  And William is currently founding and stealth, so we'll
• 2:14
  see whether he can share anything.
• 2:15
  Maybe, you know, blink twice if you have a secret to share.
• 2:20
  Maybe not yet, but HG Wells and everybody.
• 2:24
  Great. Yeah.
• 2:25
  Thanks for coming, everybody. I'm really excited about this
• 2:28
  panel, I think with large language models.
• 2:34
  We are now about a year into this kind of hype cycle and we
• 2:39
  spent a good chunk of that in what Alan Cooper, the
• 2:42
  Interaction design guide, the father of Visual Basic would
• 2:45
  call the Dancing Bear phase of technology.
• 2:49
  Where like the exciting thing is not how good the the product is,
• 2:54
  but the fact that it can do anything at all like a bear that
• 2:57
  can dance. So the the kind of theme that I
• 3:01
  want to talk about here is how do we move out of it?
• 3:03
  And in the past with with guys in the 80s, with browsers in the
• 3:08
  90s, with mobile in the 22,020. It is always required both
• 3:13
  product and design sense and technical and engineering sense
• 3:16
  to like get out of that phase into something that both like
• 3:19
  works is like technically sound but is also a good product.
• 3:24
  So to start off, let's talk about some of the successes in
• 3:31
  using LM's in products or as products.
• 3:34
  So I'm curious, maybe starting with Sarah, with your broader
• 3:38
  perspective up there on the heights of Venture, where have
• 3:44
  you seen LM's or LM powered products succeed?
• 3:48
  And what do you think for some of the key aspects that came
• 3:51
  from the design side or the technical side that made them
• 3:54
  successful? Yeah, I mean, I'd start with the
• 3:58
  premise that these are the reason that this era is
• 4:03
  different at all is because we have general technologies that
• 4:07
  are applicable to many downstream tasks.
• 4:09
  And I think what we've sort of seen in this first year is the
• 4:14
  UK capabilities we get out of them are easier to use in real
• 4:19
  environments and some tasks than others, right?
• 4:22
  And so you know, we tend to think of it in like a few
• 4:25
  different framings. One would be like how robust is
• 4:29
  the task to error. Another is just like.
• 4:34
  How much work is? In a particular task is
• 4:39
  automatable sort of end to end with the reasoning models and
• 4:44
  the generations we can do today And and so some of my favourite
• 4:48
  examples are actually in like wildly different areas right.
• 4:52
  So you have people I think like there are a lot of companies in
• 4:55
  this area now but people who are trying to automate end to end
• 5:01
  workflows in the legal world because it's a lot of text in
• 5:04
  text out right. You emailed the client, you read
• 5:07
  a document, you summarised the document.
• 5:09
  You try to reason with a particular framework.
• 5:12
  You generate legal motions. Right.
• 5:14
  Like models can do so much of this now and I think there are
• 5:18
  other professions that look a lot like text generation, right.
• 5:21
  Code is a form of text generation.
• 5:23
  We're just progressing along that dimension and another sort
• 5:27
  of field that I think surprised many of us has been sort of
• 5:31
  media creation, right. I think everybody attending this
• 5:35
  conference will have spent some time with mid Journey, but I've
• 5:37
  been super excited about the progress that's happened in
• 5:41
  video generation just because it's so, you know, useful from,
• 5:44
  it's expressive and it's useful commercially.
• 5:46
  So everything from. From runway to Peka to Hagen,
• 5:51
  which is like virtual avatars and translation that really sort
• 5:55
  of cross state quality floor over the last six months.
• 6:00
  Yeah. So you mentioned text
• 6:02
  generation, media generation and with a lot of focus on like
• 6:05
  robustness and quality. So I'm hearing a lot from you
• 6:08
  that's very much on the like technical capabilities side.
• 6:11
  I'm curious, I'm focusing on those two examples of text media
• 6:15
  generation maybe. Does anybody have any thoughts
• 6:18
  about particular product innovations that have that have
• 6:22
  like allowed those to crossover from cool tech demo to usable
• 6:26
  product that people will like, pay for or be able to, you know,
• 6:30
  deploy confidently at scale? I can talk a little bit about
• 6:34
  that. Like especially I have seen you
• 6:37
  know with all these great technologies coming up, there's
• 6:40
  one advancement which has happened is the computer, right
• 6:44
  and how we are able to compute these things.
• 6:47
  Of course GPUs are much more capable of now training the
• 6:52
  models at scale and you have all these different things.
• 6:55
  But one thing, one advancement which I'm seeing is the no good
• 6:58
  platform. So up until now, if you have to
• 7:01
  train the models, it has been like a very big task in itself,
• 7:05
  like you need to have the right kind of infrastructure in place,
• 7:08
  but also. Have to have the right kind of
• 7:10
  data set and then the whole training process itself and how
• 7:13
  long it would take on a giant is a very tedious process, right?
• 7:16
  You have to train the model from the scratch or if you want to
• 7:19
  fine tune it, from an existing model.
• 7:21
  But now there are no code platforms which are evolving and
• 7:24
  there are many, many of those. And almost every company have
• 7:27
  some variant of that. If you see right, including
• 7:30
  Amazon and Microsoft have some of that.
• 7:33
  It's now helping us. It's like becoming no brainer,
• 7:37
  the bar is becoming so low. Like OK you come.
• 7:39
  You select the base model, you select or upload your own data
• 7:43
  set, you select whatever hyperparameters parameters.
• 7:46
  You want to adjust and then boom, you have this training
• 7:49
  pipeline running in the background and now you have a
• 7:51
  model which you are able to now test it out.
• 7:53
  So I'm really seeing that as a breakthrough in the industry.
• 7:58
  Because that's helping anyone and everyone to come and try out
• 8:01
  a variant and then see how the model is working.
• 8:04
  That is also opening up opportunities in the community
• 8:06
  to have lot more OSS models out there for people doubt and
• 8:10
  explore and you know, building that infrastructure and
• 8:12
  ecosystem. Yeah, ohh did.
• 8:17
  You want to take over? I just wanted to add on that as
• 8:20
  well that one of the things you basically ask what enabled this
• 8:23
  revolution in many ways everybody to have these larger
• 8:26
  datasets of text. For example, GT3 was trained on
• 8:29
  like Common Crawl and this kind of data set.
• 8:31
  So having more data also well curated, you want one of those
• 8:36
  catalysts to what we see today with all this amazing network
• 8:38
  models? Definitely from a.
• 8:42
  Yeah, from the commercialization standpoint, I mean, so ignore
• 8:45
  the big players, right, like GitHub, Copilot, major money and
• 8:49
  just check GPT, they're low level as well, at least in
• 8:51
  shafts guys. It's a low level abstraction,
• 8:53
  but it is. Such a powerful tool, and if you
• 8:58
  learn how to prompt it, you've seen a large amounts of money
• 9:02
  flowing into these services. Of course could have copied has
• 9:06
  many years head start in front of many, many other folks, but.
• 9:11
  Yeah. We're obviously seeing a lot of
• 9:13
  new products being developed, but those are the obvious ones.
• 9:17
  Yeah. And I think it surprised some
• 9:20
  people that ChatGPT was such a huge explosion relative to the
• 9:24
  GBT 3 API. But I think if you've sat a
• 9:28
  non-technical person down in front of something like the
• 9:31
  playground or the API versus the chat interface, it's like it's
• 9:36
  night and day. So I'm wondering maybe from
• 9:40
  Shamala, you've worked a lot on voice assistance in particular.
• 9:43
  So I'm curious what have you seen that are the kinds of like
• 9:48
  unintuitive or surprising or high leverage changes to these
• 9:52
  voice assistants that it's not necessarily just about their
• 9:55
  engineering? Abilities, their latency or
• 9:56
  whatever, but the actual sort of like product design, the whole
• 9:59
  product thinking that makes them sticky or better.
• 10:02
  Yeah, I mean if you look at and that's a very good question.
• 10:05
  So if you look at the traditional way of designing, I
• 10:08
  mean in my mind when you're designing a digital assistant, a
• 10:12
  lot of different generating high capabilities are there.
• 10:14
  I mean I don't like to call it like conversational AI, but I
• 10:18
  like to call it generating. I because ASR is a kind of
• 10:21
  generative AI speech to text or text to speech is.
• 10:24
  So digital assistants in general, the traditional way,
• 10:27
  the way to work was this some sort of speech recognition
• 10:31
  system which would you know. Interpret what you are saying.
• 10:33
  And then there was the understanding that is where you
• 10:36
  know a lot of advancements has happened.
• 10:38
  We used to have an AP and it was very much rule based.
• 10:41
  You had intent based and you would have a couple of
• 10:44
  utterances tied to an intent and you would say like OK, if the
• 10:47
  user is asking this, these are the different utterances and it
• 10:49
  would not go beyond that. So there was some advancement
• 10:53
  which happened eventually. Like yeah, so for example, will
• 10:57
  it rain in Seattle? Will I need an umbrella on
• 10:59
  what's the weather in users mind they all meant the same thing,
• 11:02
  but it was not very natural for us to or the language models to
• 11:06
  have that kind of capability and we had to, you know?
• 11:09
  Into each of these different utterances.
• 11:11
  I still remember those days when I used to collect these
• 11:14
  utterances from users through different mechanisms and then
• 11:17
  feed it into the system and train the model.
• 11:19
  That has changed. So large language model in my
• 11:22
  mind is like a rapper around the conversational AI technology,
• 11:27
  but in an advanced way. So now it's so much open, it's
• 11:30
  able to understand because it has so much, you know, knowledge
• 11:33
  from different sources. It it is able to interpret the
• 11:36
  intent without it having to, you know, manually feel the intent
• 11:40
  now. The other advancement which has
• 11:43
  happened, so when I used to work for Ford Motor Company, one of
• 11:46
  the challenges we had but then was we had this huge owner
• 11:50
  manual, right. And then if we have to voice the
• 11:53
  party or voice enable or make these, you know, owners manual
• 11:58
  as a conversational assistant thing, I can chat about it or I
• 12:01
  can talk about it. Believe it was very complicated
• 12:05
  with all the things like we had a lot of manual effort band
• 12:08
  there to be able to create a bot out of the knowledge base out of
• 12:13
  that owners manual. Now with the advancement with
• 12:17
  the rag and they all these different things coming
• 12:20
  together, I feel like it is becoming much more easier.
• 12:23
  You are able to feel all these different knowledge sources to
• 12:26
  the system and it is able to you know generate the answers based
• 12:30
  on that. So that is really going to be
• 12:33
  the next big thing, especially in the digital assistant world.
• 12:36
  Because it takes away the effort of someone having to manually do
• 12:41
  all the effort, which was the case back in like a couple of
• 12:44
  years. You know, I I think while we're
• 12:48
  talking about assistance, I think it is like Sam Altman who
• 12:52
  said like we're trying to replicate the median human and I
• 12:55
  think in the Voice Assistant space or generally with
• 12:59
  assistance and and like a personified box, it's not clear
• 13:04
  to me people want the median human, right?
• 13:06
  They want either the best M and a lawyer or they want Elon Musk
• 13:12
  or they want their personalised version, right.
• 13:15
  They want like their AI. And so, you know, I think one of
• 13:18
  the things that has been really interesting to see, you know,
• 13:22
  piggybacking on. What was just said is there's so
• 13:26
  much to do in terms of development of empathy, right?
• 13:29
  Like media richness, like voice and video interrupts fluidity of
• 13:34
  speech. And then also on the sort of
• 13:36
  more. Model development side in terms
• 13:40
  of like personalization memory skills like things the
• 13:42
  assistance can can do and so you know I I'm an investor Inflexion
• 13:48
  and also great companies like character out there.
• 13:50
  I I I don't know that we're going to end up with median
• 13:53
  human versus like experts and personalization and like you
• 13:57
  know, humans are interested in individuals, right?
• 14:01
  I think there's a tendency to say it's a median human because
• 14:05
  the models hallucinate and so we're trying to set our
• 14:07
  characters expert, and expert can judge the quality of the
• 14:10
  system. But really, I think so.
• 14:12
  You're right, there's a new clause of product that can still
• 14:16
  fulfil the job to be done, but it sets a legal team and they've
• 14:19
  got some problem. But what you want is a system
• 14:23
  that can scan millions of documents where human couldn't
• 14:25
  have done that before. So the breadth and perhaps even
• 14:28
  the depth can go and find correlations between documents
• 14:31
  that a human would never be able to find.
• 14:33
  So you'd be augmenting your team and unlocking new possibilities
• 14:37
  with these products. Yeah, actually exactly about
• 14:40
  that. What I usually see with like
• 14:42
  million human kind of Elema applications and really great
• 14:45
  applications, everything around evaluation.
• 14:47
  So they have the capability already.
• 14:50
  It's mainly they're on vacation that's being written and written
• 14:53
  up way which then leads to elimination.
• 14:55
  So teams that do it really well usually have a whole evaluation
• 14:59
  set, a whole data sets that allow them to constantly monitor
• 15:02
  and make sure that the performance they see is actually
• 15:05
  what they expect and then iterate on that.
• 15:06
  And that's what usually takes from a medium human health level
• 15:10
  to an expert level. So it seems that this
• 15:17
  personalization with the like successive character dot AI for
• 15:19
  example, this like personalization is clearly like
• 15:23
  a pattern that can be applied in other places.
• 15:26
  I'm wondering if people have any thoughts on other patterns or
• 15:31
  anti patterns that they've seen in the creation of like NL and
• 15:35
  powered feature or an LM powered product.
• 15:38
  Yeah, I think that everything that involves latency is a
• 15:42
  really challenging for everyone. So I mean probably the one of
• 15:46
  the hardest intersection is latency and user facing and
• 15:50
  open-ended. So many companies building
• 15:52
  open-ended chat or interfaces that behind the scenes we have
• 15:56
  empowered apps. It's really difficult to make
• 15:57
  sure that we see the experience or we have the experience we
• 16:00
  expect to have. So again having this relation
• 16:03
  datasets really helps but also latency the big challenge so how
• 16:07
  do you make sure you protect all anyone to get great performance.
• 16:10
  So you may have like multiple prompts and multiple like
• 16:13
  chatter for indication. Doesn't matter and the like, but
• 16:17
  you also want to make sure the experience it really fast.
• 16:19
  So if you have let's say a chatbot, I've been Stripe, we
• 16:22
  launched this striped docs that allow you to ask questions
• 16:25
  sometimes for the communication. Then how do you make sure that
• 16:27
  the experience is fast enough but at the same time you don't
• 16:30
  see illuminated text. So it had one of those examples
• 16:34
  with Sydney when they just launched.
• 16:36
  People would play with the chatbot and then they got all
• 16:38
  these awful results sometimes And then after a few weeks or
• 16:42
  thinking days they deploy this update that after you get a
• 16:46
  generation there is another prompt running behind the scenes
• 16:49
  asking the generated answer is actually not harmful.
• 16:52
  Biassed and the like. And if so, it literally like
• 16:54
  deletes the answer. So many ways and it's like
• 16:57
  latency and also maintain performance, but it's definitely
• 17:01
  a. Challenging space, yeah.
• 17:04
  I think that particular example is one I saw a lot of people
• 17:07
  pretty upset about. To see an answer appear and then
• 17:09
  like suddenly disappear feels a little bit dystopian.
• 17:12
  So that's one of my favourite examples of an anti pattern for
• 17:14
  out to solve that problem with the latency versus correctness
• 17:18
  and check. Yeah, maybe I'll I'll mention
• 17:22
  two different patterns I think are are kind of interesting ones
• 17:27
  in a company that we invested in and one is just something that
• 17:30
  like we're continually looking for that we we think will work.
• 17:33
  So umm a, a common issue is that humans are like not that good at
• 17:41
  prompt writing or specifying exactly what they're looking
• 17:44
  for, right. And so an example I'd use is
• 17:47
  where investors in this company seek dot AI they.
• 17:52
  Use among other technologies like SQL generation from natural
• 17:56
  language to enable business users to ask questions of
• 17:59
  structured data sources like the warehouse.
• 18:02
  But as you might imagine, like any company that has a
• 18:05
  reasonable amount of data, 16 data teams, the idiosyncrasies
• 18:10
  of like how they actually transform their data and like
• 18:12
  what you know, customer account means in this department.
• 18:15
  This is the other department means it's actually very hard to
• 18:19
  go from a general usually under specified question from a from a
• 18:23
  business user to the right generation right.
• 18:26
  Like the sequel is actually not like we can be best.
• 18:29
  But that is not actually the hard part.
• 18:32
  So the pattern I think is kind of interesting is companies
• 18:36
  figuring out like what is the workflow to be predictive,
• 18:41
  right? Like use for example like, you
• 18:43
  know, query generation or ranking technologies or
• 18:46
  iteration with the end user to get better specification about
• 18:49
  what they're trying to do versus expect every end user to be good
• 18:53
  at. Well, essentially prompt
• 18:55
  engineering, right. So I think that's 11.
• 18:58
  Interesting. Sort of a frontier that people
• 19:02
  are working on. Another is like everybody loves
• 19:08
  Copilot, but Copilot is at once like one of the most like very
• 19:14
  limited and also crazy ambitious things to try to do right.
• 19:18
  Like we work with Alex Gravely who's on to a new mesh in which
• 19:21
  and he's the chief architect at at Copilot and he was like well
• 19:24
  like the the dominant feature. We've talked a couple of times
• 19:26
  about latency but it was like you know if I type 100 words per
• 19:30
  minute like you got to stay ahead of me to be useful like
• 19:32
  that's a pretty tight latency bound for anything for anything
• 19:36
  that is LM powered. And that also means like you
• 19:41
  know you're you're doing like. I think almost synchronous
• 19:44
  autocomplete with local contacts like that is not all we need to
• 19:47
  do in in software engineering and so particularly in these
• 19:51
  areas where. There's been some you think
• 19:55
  it'll there's like going to be success in the modality.
• 19:58
  I think the the interest in trying to do more things
• 20:02
  asynchronously and figure out how to come back like go run a
• 20:05
  job, right. You know if you're an engineer
• 20:07
  you work my team like you go do something and you come back to
• 20:10
  me and say like I figured it out or I didn't or I have a question
• 20:13
  right. And so I I think that there's
• 20:15
  both like product and research effort around how can I sort of
• 20:20
  use the the to some degree the scalability that William
• 20:24
  described to go try more things, do test time, search like you
• 20:28
  know try and compile commands and come back.
• 20:30
  And and tell a user like, ohh, I did, I did more for you and it's
• 20:34
  like let me notify you that something's happening or gather
• 20:36
  more data. And I think there's gonna be
• 20:38
  more emphasis on sort of the more asynchronous patterns as
• 20:43
  well. I think a different way to put
• 20:47
  that is where do all aims really annoy people and suck.
• 20:51
  It's put them on the critical path and you put them
• 20:53
  synchronously in front of the user on the job that they're
• 20:57
  familiar doing. But whether it's writing code,
• 20:59
  there's a kind of code. While it's on the side, it's
• 21:00
  additive. It comes in and makes a
• 21:02
  suggestion, but you can quickly glance and say you know you're
• 21:05
  crazy, I'm going to ignore you and I'll continue doing my job.
• 21:09
  They got applies to almost all use cases, like humans have a
• 21:11
  specific way of doing things and you just incrementally introduce
• 21:15
  yourself. And then as these copilots move
• 21:18
  closer to pilots, you know, we'll defer more to them.
• 21:21
  But for now, they're still hallucinating.
• 21:23
  And they're very slow. And so as that improves will you
• 21:26
  know hand over, hand over more. Yeah, I think a lot of the time
• 21:32
  for that kind of asynchronous communication or that ignorable
• 21:35
  communication, it's like text based or visual, you know, it's
• 21:38
  a it's a toast that says, hey, maybe you want to do this, or
• 21:41
  the ghost cheques in Copilot. Relatively sorry, one.
• 21:45
  More point is like. The the fatigue and I think
• 21:47
  we'll sit in. So even in Co generation if as
• 21:50
  humans we're going to constantly be reviewing the agents.
• 21:53
  So this is the tech code generation like that can become
• 21:57
  very overwhelming to engineers, right.
• 21:59
  Like it could become taxing And so you want that accuracy bar to
• 22:03
  be high, or you want to introduce the agent in a way
• 22:05
  that is. Robs reviewing your code and not
• 22:07
  you reviewing its code. Umm, yeah.
• 22:12
  If you 10X the amount of code to review, you 10X the amount of
• 22:14
  code review to do, which people don't talk about when they talk
• 22:17
  about boosting productivity. So the question I was gonna ask
• 22:23
  to Shamala was with audio, that's like something that
• 22:27
  really captures your attention. It's like, you know, even a tiny
• 22:29
  glitch in audio is like something that you just
• 22:31
  immediately snap onto. I'm curious, do you have any
• 22:33
  thoughts about how this sort of like asynchrony?
• 22:37
  For this, like you know, Deferral might work with
• 22:40
  something like a Voice Assistant.
• 22:42
  Does it have like can I call you on a cell phone or or how?
• 22:46
  How else can they achieve that? So your voice kind of broken
• 22:50
  between, if you could just repeat it.
• 22:52
  I don't know what's happening with my Internet, so yeah.
• 22:54
  Incredible and audio issue while talking about the.
• 22:57
  Troubles exactly the audio issues that the language talking
• 22:59
  about. Yeah, so text assistant can pop
• 23:03
  up a notification or do ghost text and that allows it to fade
• 23:06
  into the background. But for audio, it's not obvious
• 23:10
  to me how you would do that. And I don't think I've seen that
• 23:11
  pattern with anything like, you know, Alexa just like interrupts
• 23:14
  my family dinner if I happen to say its name.
• 23:17
  So curious if you thought at all about how that like asynchrony
• 23:20
  or or? You know threading into the
• 23:23
  background and Co piloting might work for audio.
• 23:27
  So audio is just one part of it. I I'm glad you brought it up
• 23:31
  because I talk about some of these things in my book,
• 23:33
  emotionally engaged digital assistant, right.
• 23:35
  Like when I'm trying to bring bring all of these things
• 23:37
  together, understanding personalization.
• 23:40
  So in my mind, you know, it's it's all about affordances like
• 23:44
  how it should not always be proactive.
• 23:47
  Like you know, every time I'm seeing Alex, I'll do this, you
• 23:49
  know it will do it right. But in certain cases it is very,
• 23:53
  you know, proactive in the sense like if something, you know,
• 23:55
  it's giving me gestures, there's multimodality even in that
• 23:59
  context you have like different devices.
• 24:00
  So I feel like that's very important.
• 24:04
  How the feedback on notification is coming when I'm doing a
• 24:07
  certain task and especially with all the different kind of
• 24:10
  automations I'm able to do around, You know, for example,
• 24:15
  I'm able to send the entire schedule for my day saying you
• 24:20
  know at this time I want this to happen, at this time I want this
• 24:22
  to happen and then you know like set up an alarm, turn on my AC
• 24:27
  or whatever, you know, I'm also setting up my routine nowadays.
• 24:29
  How does that happen? Hmm, right.
• 24:32
  How is the user getting notified the job is done?
• 24:35
  Or how does the user know what's happening?
• 24:37
  Sometimes there's audio cues, sometimes there's visual cues,
• 24:40
  sometimes there's some notification, sometimes it's
• 24:43
  just like, you know, everything done now and this is the task.
• 24:46
  So I feel the same kind of pattern will be leveraged in the
• 24:51
  world of Co pilots as well. You mentioned multimodality in
• 24:57
  your response and that's something that like looking a
• 25:00
  year or two ahead once GPT 4V is generally available and once
• 25:05
  other proprietary model providers and open models have
• 25:08
  caught up and capabilities the clear big win has.
• 25:13
  Does anybody have any examples of products that take advantage
• 25:16
  of that multimodality that they have either seen, seen a demo of
• 25:22
  that they want to share? A product called Neva, the Nemo
• 25:26
  Vision Assistant it is. It is a multi modal model and.
• 25:30
  Open the Eye just launched. I'm going I think with the as
• 25:32
  well. So with this model it's
• 25:35
  available for trying out. You can go to our playground in
• 25:38
  Media I playground and try it out as well.
• 25:40
  What this does is it's an image and text understanding model.
• 25:45
  So it's not just generating all the things what we spoke about
• 25:48
  like text to pigs, but it is able to take an image and it is
• 25:51
  able to also if I have uploaded an image and I asked it like hey
• 25:55
  describe that image, the model is able to describe the image
• 25:58
  for me. If I just give the image and say
• 26:01
  write a story based on the subject of the image or the
• 26:04
  patron of the image, the model is also able to do those things.
• 26:07
  Tonight and the London is able to work stand alone in the sense
• 26:10
  like I can just have text and I can use it as a LLM any any text
• 26:15
  to text model or I can use it in the combination with text and
• 26:18
  image and it is able to generate and give me responses based on
• 26:21
  whatever the subject line of the image might be.
• 26:24
  That is one level of amending modality which really is
• 26:28
  interesting. I don't think about
• 26:30
  multimodality like yeah, I mean just generating an image kind of
• 26:33
  context, but I think about it more on the utility standpoint
• 26:37
  like how it is helping the users, right.
• 26:40
  And that's very interesting because.
• 26:43
  Think about people who have accessibility issues.
• 26:46
  If right now everything which we have, everything which we have,
• 26:51
  even when I'm posting on LinkedIn or somewhere else, I
• 26:53
  have to add the alt text so that people who are using screen
• 26:57
  readers, they are able to actually understand what I
• 27:00
  uploaded. Otherwise they will not know
• 27:02
  what's happening. But multimodality can change the
• 27:05
  dynamics in that context. I don't have to the elements
• 27:11
  capable of doing that for me. So it's really descriptive.
• 27:14
  So even a person who have any kind of disability or want to
• 27:17
  understand what's happening in the context, what's the image
• 27:19
  and all, they are able to get much more information than an
• 27:23
  all time which I will write which will be like a person
• 27:26
  sitting on the bench. That's it, right?
• 27:29
  That is the kind of thing. So I'm thinking about the the
• 27:32
  multi modality in that context and it feels great what it can
• 27:37
  do for the advancement of the industry or the community.
• 27:42
  Yeah I'm also adding on that maybe like the first part is
• 27:45
  about what does it mean for like product and engineering and
• 27:47
  designing that sense. So the first one unlike maybe
• 27:50
  text, we should remember that multi modality usually more
• 27:53
  video, audio which are more expensive to compute also store
• 27:57
  which is going to change file the profit margins and also
• 28:00
  latency constraints it's are already quite challenging even
• 28:02
  with text at the at the modality.
• 28:05
  And then the kind of product that I've seen doing it
• 28:07
  relatively well. So I think multimodality double
• 28:10
  well is one that you don't really feel There is like
• 28:12
  multiple modalities behind the scenes.
• 28:13
  So easy in the way Google search interactive search experience is
• 28:17
  one example where you. Right text or prompt and you get
• 28:20
  multiple results based on video and image behind scenes and then
• 28:24
  two areas I think that are currently are going to explode
• 28:27
  once multimodality gets there. I mean the first one is
• 28:29
  everything around content generation like imagine we can
• 28:32
  create TikTok videos like seamlessly if we use 11 laps or
• 28:36
  combine it with soda and all these other amazing products to
• 28:38
  generate our videos without all the efforts currently associated
• 28:41
  with that. And the second part is, I think
• 28:44
  Charles, we talked about a few weeks ago with everything around
• 28:46
  robotics, right. So robotics is also having its
• 28:48
  moment. So once you can combine like
• 28:50
  vision and and other sensory and signals you can do some amazing
• 28:55
  stuff with robotics as well. I I think one of the things
• 29:00
  going back to making it easier for it, it's it's not
• 29:05
  necessarily focused on multi modal understanding in these
• 29:10
  sort of final more extreme sense.
• 29:13
  But the ability to take richer user input versus expecting
• 29:17
  everything to be taxed I think is really powerful, right.
• 29:20
  And I'll give you a few examples and Sahar just made me think of
• 29:23
  this right if you and being very use case specific is is very
• 29:28
  impactful. So for hey Jen, like you, you
• 29:32
  could as a user like string together a bunch of things and
• 29:35
  say like OK, I'm going to use element.
• 29:36
  I'm going to use 11 to, you know, take do speech to text and
• 29:42
  then do text to speech in a different language and then like
• 29:44
  try to match it up with the sort of video avatar models that
• 29:51
  allow us to match like lip movements and motions, right?
• 29:55
  Or a product company could do that for you, right?
• 29:58
  And there's definitely research involved in designing models to
• 30:01
  interact well that way. But I think the fact that now
• 30:05
  like what users want to do is just give you a video in one
• 30:07
  language and get video out in a different language, that works
• 30:10
  perfectly. I think that's one example.
• 30:13
  Another example would be something like Pika being able
• 30:16
  to, you know create video from a given image is, is it allows for
• 30:25
  much simpler controllability than right?
• 30:28
  And we've also seen people figure out how to go back and
• 30:31
  forth with editing that. So like you know, managing
• 30:35
  structure in images and videos is still very challenging,
• 30:38
  right? So a smart product person way to
• 30:40
  go handle this well, integrate with workflow tools that allow
• 30:43
  you to manage structure, right? That could be a a Photoshop or a
• 30:48
  3D3D modelling tool, change the structure in a very specific way
• 30:55
  and then be rerender for example, right.
• 30:58
  And so that I I think there are lots of interesting product
• 31:03
  innovations happening right now to allow people to end users to
• 31:07
  get the outputs they want and more intuitive ways, which
• 31:10
  includes like editing and input in different modalities.
• 31:14
  Right. I think that's all the time that
• 31:17
  we have for this panel. So I wanna thank Shamala, Sahar,
• 31:20
  Will and Sarah. Really great conversation, a lot
• 31:22
  of very distinct but complementary perspectives on
• 31:26
  how we're gonna make useful products out of large language
• 31:29
  models and large multi models, multimodal models in the future.
• 31:32
  Thanks a lot. Thank you very much everybody.
• 31:35
  And Charles, thank you for being such an excellent host and
• 31:40
  moderator. Yeah.