Latest / Agentic Conversations (formally mlops.community) / Ux of an LLM User Panel // LLMs in Production Conference Part II

Transcript

Ux of an LLM User Panel // LLMs in Production Conference Part II · Agentic Conversations (formally mlops.community)

• 0:00
  What's up team? Before we get started on this
• 0:02
  podcast, I want to mention that we are doing another Elms in
• 0:04
  production. This is the third one, and you
• 0:06
  know how it goes when you iterate on something.
• 0:09
  It just keeps getting better. We've got some incredible
• 0:12
  speakers, all new speakers. I think there's only one or two
• 0:15
  that are coming back from a previous edition of the LMS and
• 0:20
  production. There's just so much good stuff
• 0:23
  going on out there and we want to highlight it.
• 0:25
  Go and sign up now. The link is in the description
• 0:28
  or you can go to home dot jobs dot community, check it out.
• 0:32
  We're going to be talking about all different kinds of stuff
• 0:35
  ranging from deploying open source models, regulatory
• 0:38
  aspects of LMS, how to build economical LLM solutions, how to
• 0:43
  select prioritise LLM use cases, modes in building LLM solutions,
• 0:48
  how not to be a GBT wrapper and engineering best practises in
• 0:53
  LLM solutions. Shout out to our main sponsor
• 0:56
  process for making this all happen.
• 0:58
  We wouldn't be doing it without them.
• 1:01
  And I would love to see you there because you know what I'm
• 1:04
  going to do? I'm going to be playing some bad
• 1:08
  musical interludes in the breaks.
• 1:11
  I can guarantee that. I also can guarantee that the
• 1:14
  quality of talks you're going to get, you cannot find anywhere
• 1:17
  else on the Internet. That's all.
• 1:19
  Enjoy this podcast? All right.
• 1:26
  Our next panel is an awesome group.
• 1:29
  We have Misty from Jasper, AI, We have Davis from Innovation
• 1:33
  Endeavours, We have Dina from Adobe, Firefly and Artem, CTO
• 1:38
  Verdine. And they're all coming together
• 1:40
  to chat about UX of LM users. So without further ado, let's
• 1:45
  bring them up to the stage. Hello, Davis.
• 1:50
  Hello, Misty. Hello, Arda.
• 1:54
  And last but not least, Dina. All right.
• 1:59
  Thank you guys so much for joining us.
• 2:01
  Yeah. Thank you, Willie.
• 2:02
  Are we good to get started? Yeah, go ahead.
• 2:05
  Alright, hi everyone, I'm the moderator for this sale today.
• 2:09
  Today we're going to focus on UX and LM, which is a really.
• 2:11
  Interesting space. And so maybe to start, Misty,
• 2:14
  Artem, Dina, maybe we can go around quickly one by one and
• 2:16
  you can each briefly introduce yourself and in particular talk
• 2:20
  about some of the recent. LM features you've worked on and
• 2:23
  I'll dive into some more specific questions from there.
• 2:25
  You want to go first. I'm Misty.
• 2:29
  I'm a product manager at Jasper and Jasper is really focused on
• 2:33
  supercharging content marketers especially and content marketing
• 2:37
  teams with their content creation.
• 2:39
  So a future that we recently rolled out and I actually just
• 2:43
  got off of a a webinar introducing it to customers is
• 2:47
  Jasper campaigns. So we've introduced that ability
• 2:51
  with just one set of context with your selective brand voice
• 2:54
  to create all of the assets that you need for a campaign.
• 2:58
  Instantly and be able to add anything really quickly with
• 3:01
  that. So that's been a big thing that
• 3:04
  we've been really focused on at Jasper.
• 3:06
  That and just in general making content creation as easy and
• 3:11
  seamless and not a headache for users as possible.
• 3:14
  Awesome. Our time.
• 3:15
  You want to go next. Thanks, Davis.
• 3:18
  I'm cofounder, CTO at Bardeen AI.
• 3:22
  We're a platform for. Creating web web browser based
• 3:28
  automations with heavy use of AI across the board.
• 3:33
  So literally today the launch that we did was we launched our
• 3:37
  ChatGPT plugin and basically what it does is it allows us to
• 3:43
  build automations using natural language and this is kind of
• 3:48
  built on top of feature that we have inside the product which we
• 3:51
  call Magic Box where you just type what you want to do and we
• 3:56
  create an automation. For you on the fly, great.
• 3:59
  Hi, I'm Dina, I'm product manager at Adobe, working on
• 4:02
  fire flights of family of generative AI models for
• 4:06
  creatives. It does text to image, text to
• 4:10
  fonts. I mean it's it's a model, It has
• 4:13
  its own website where you can, you know generate different
• 4:17
  pictures. But we also recently integrated
• 4:19
  Firefly into different flagship products with Adobe.
• 4:22
  So it's now powering things like Adobe Photoshop generative feel.
• 4:27
  It's on. Illustrator, generative recolor
• 4:30
  and then it's also an Adobe Express and hopefully more
• 4:33
  products. So, so yeah, excited to be on
• 4:36
  this panel. Disclaimer, not a UX designer,
• 4:39
  so you know they're probably better people to talk about UX
• 4:42
  design. Self.
• 4:43
  I'll try my best. And also yeah, just representing
• 4:47
  mainly my experience working on this both inside, outside of
• 4:49
  Adobe, so not representing any product, not talking about any
• 4:53
  product road map, just my own views.
• 4:56
  Awesome. And so maybe to start, I thought
• 4:58
  it would be good if you could each go one by one and talk
• 5:00
  about what's the single biggest UX challenge that you face.
• 5:04
  And kind of building the product that you just described?
• 5:06
  And maybe you could. Talk through different things
• 5:08
  you've. Tried and.
• 5:09
  Ultimately, what you had to figure out in order to solve
• 5:11
  that UX piece of the product you've built.
• 5:13
  And so, Misty, why don't we start with you on that?
• 5:15
  Yeah. So like I talked about at
• 5:19
  Jasper, we're really trying to solve especially the content
• 5:24
  creation pain point for marketers.
• 5:26
  And we are extending outside of that into more of the research
• 5:29
  and the ideation that happens before content and then more of
• 5:33
  the publishing and distribution that happens after.
• 5:36
  But just within that content creation piece we're seeing, and
• 5:40
  I'm sure everyone is feeling this, the fatigue of so many
• 5:44
  tools coming out and so many different ways.
• 5:46
  Using AI and you can do anything from using chat to using like an
• 5:53
  extension like Air PRM to get a prompt.
• 5:56
  And so there's so many different paths and I think everybody went
• 5:59
  through this phase of being so excited and enthusiastic to try
• 6:03
  anything and everything and now people are starting to feel more
• 6:06
  and more of that fatigue. And so the big challenge that we
• 6:10
  have is, is continuing to to try and deliver the most efficient,
• 6:16
  delightful way for people to. Solve those content creation
• 6:19
  problems while balancing that fatigue of another new thing
• 6:23
  that I have to try. And so is that something that
• 6:27
  that we're always working on and always making sure that our any
• 6:30
  new features that we do are really well validated with our
• 6:33
  existing customers and with the market in general to make sure
• 6:37
  that there's an appetite for it and that we're not just further
• 6:39
  contributing to that burnout of new features.
• 6:42
  Our time. What about you?
• 6:44
  I think unpredictability and nondeterminism of LM's is a is a
• 6:52
  UX is. That is definitely a challenge
• 6:54
  for users in terms of experience because you pretty much never
• 6:58
  know what you're going to get. Sometimes very simple things do
• 7:02
  not work and then very sophisticated things work
• 7:06
  surprisingly well. I kind of want to echo what Mr
• 7:10
  said in terms of fatigue and hype, you know, around this.
• 7:14
  So I think at this point. Users expect a thing that would
• 7:19
  work. And so for us, the challenge was
• 7:21
  to embed this new amazing kind of capability that LM's provide
• 7:27
  into the into the flow of our product and make it seamless and
• 7:32
  make it so that it really makes user experience better as
• 7:37
  opposed to just your being there.
• 7:39
  Because everyone does that and because it's a cool thing to do.
• 7:42
  So for us, the main challenge was to kind of.
• 7:46
  Design the experience around the fact that.
• 7:52
  You know LM's are not perfect. And come up with the modality,
• 7:56
  come up with a way that would that like basically almost make
• 8:01
  it that so users don't notice it right.
• 8:04
  And if you think about it, a lot of amazing products early days
• 8:07
  had the same problem like Google.
• 8:09
  You know with the very first versions you almost never got
• 8:12
  your results right and they even.
• 8:15
  But I think the way they solved it is with with the UX.
• 8:18
  So when you search something, they drop you on the page with
• 8:22
  the results, but the search box is still on top.
• 8:25
  And so like naturally, if what you're looking for was not
• 8:28
  there, you would go and refine that search.
• 8:30
  And later as the product matured, they started doing a
• 8:34
  better job at it. So where now when I'm starting
• 8:36
  to type there like 5 things at the drop down that I can click.
• 8:40
  But it's essentially the same thing.
• 8:42
  It's this kind of a play between, you know, your model
• 8:46
  and your user where like they they do this like sequence.
• 8:51
  And I think that the biggest challenge is how to make this
• 8:53
  sequence seamless. So it's like.
• 8:55
  Very natural to the human being who is in front of the computer
• 9:00
  using your product. And we may have just lost DNA
• 9:02
  for a second. And so maybe a quick follow up
• 9:04
  on that. Can you talk through some of the
• 9:06
  core design patterns that you built out that kind of helped
• 9:09
  you solve that problem? So I think one thing you touch
• 9:11
  on for example is refinement workflows.
• 9:13
  You know the LM produces an initial.
• 9:14
  Response you allow people to edit.
• 9:17
  What are different things you've tried to make that work and
• 9:19
  like? Any learnings?
• 9:20
  On what? Worked well when you're doing
• 9:22
  that and what? For sure.
• 9:23
  Yeah. I think one thing that I mean we
• 9:25
  literally did what Google is doing right so now and like if
• 9:29
  you go into our product and try to generate you know an
• 9:33
  automation from your description, we will kind of
• 9:37
  drop you into the preview mode and we still have that that box
• 9:40
  on top. So users naturally kind of, you
• 9:44
  know, know, there's like, oh OK, this is not exactly what I what
• 9:47
  I was looking for. Let me go and refine that.
• 9:51
  The other thing I think is kind of like nudging users to give
• 9:54
  them their right. Idea of like how to formulate
• 9:58
  their query. So basically think of it as like
• 10:01
  a a drop down with suggestions of like what might be relevant.
• 10:06
  And I think it's the interesting challenge there is to kind of
• 10:10
  use existing like old school AI, things like recommender systems,
• 10:15
  collaborative filtering, to kind of find the right things to
• 10:19
  suggest to the user that they can go and further like use with
• 10:24
  an LM. And it's a fascinating thing
• 10:26
  because. If you think about it, the.
• 10:30
  Effort to go from like nothing to a 3040% working prototype is
• 10:35
  virtually 0. Like you can in an afternoon you
• 10:38
  can get something that get like and then you post it on Twitter.
• 10:43
  You get excited, you slap a waiting list on top and like you
• 10:47
  have your new start up. But then you hit this almost
• 10:50
  like a vertical wall of like you know if you want because that's
• 10:55
  not a product. Like you know, if your product
• 10:57
  only works 3 out of 10 times, users are not happy with it.
• 11:01
  They don't care if it's new technology so and then to make
• 11:04
  this improvements further then it becomes like really
• 11:07
  challenging and you have to find this new way like creative ways
• 11:11
  to to actually kind of overcome this and you know have other
• 11:16
  things help you help your users use the LM in the right way.
• 11:21
  That makes a lot of sense. I think both of you touched a
• 11:24
  little bit on what I would. Describe as like the broader
• 11:26
  idea of interfaces, right? So natural language is obviously
• 11:29
  a more common. Interface with LMS in.
• 11:31
  General, even natural language. There's different ways to
• 11:33
  express it from chat to maybe just a one shot language
• 11:36
  command. And then a lot of teams are
• 11:38
  actually moving away from language at all and LM and you
• 11:41
  try to abstract it behind, you know, a button.
• 11:43
  I'd be curious how you guys have thought about interfaces for LM
• 11:47
  based products and how you've learned or thought through like.
• 11:50
  What is the right interface for these different types of
• 11:52
  features? We missed you.
• 11:53
  We can start with you. Yeah.
• 11:56
  And yeah, I can definitely relate to that.
• 11:59
  Like overnight we have so many engineers all the time, the old
• 12:02
  build, the POC and we all get really excited like wow, you
• 12:06
  could do that in a night. But yeah, then you hit that
• 12:08
  vertical class. So we have kind of been all over
• 12:12
  the map and Jasper and have had a lot of internal debates and
• 12:16
  have done a lot of testing over whether the chat interface and
• 12:21
  like leaving it totally free form is better or we.
• 12:26
  Actually just this morning obscured our command bar, which
• 12:31
  our command bar was like a very loved, very cool thing where
• 12:35
  within a document editor you had a bar where you could type in
• 12:38
  commands and generate outputs and with one click add them to
• 12:42
  your document. And after a few months of use,
• 12:47
  we realised that it was clunky and it was in the way.
• 12:49
  So it's something that we're always evolving and iterating on
• 12:53
  the product. I talked about at the beginning
• 12:55
  of this Jasper campaigns that started off with a chat
• 12:58
  interface with just your typical bot interaction of Tell me about
• 13:03
  what you're promoting, tell me about what your goals are and
• 13:06
  who's your target audience, and we.
• 13:10
  After some feedback, decided that that was even too
• 13:12
  difficult, that having to read and to write so much in order to
• 13:17
  get to that path of content generation was too cumbersome
• 13:19
  and therefore move to more of that button interface.
• 13:22
  So it is something that we are going to be evolving and
• 13:27
  changing every day. Like we took out the command bar
• 13:30
  this morning, Maybe we'll bring it back today, I don't know.
• 13:32
  But we will always get that feedback, and a lot of that
• 13:36
  feedback is coming through what we're seeing people actually use
• 13:39
  which will. See by often what they're
• 13:42
  copying out of their outputs and taking someplace else.
• 13:45
  But yeah, always do you have. Any I'm curious, given you've
• 13:48
  tested so many things, do you have any intuitive?
• 13:51
  Rubrics or rules? For when you think, oh, this
• 13:53
  type of problem space. This type of feature lends
• 13:55
  itself more. Towards this type of interface
• 13:57
  or is one of your learnings that in a lot of these cases you just
• 14:00
  need to test them all and see and it's hard to know a.
• 14:03
  Priori, what's going to be the best?
• 14:05
  Yeah, it's really hard to know what's going to work best in any
• 14:09
  situation. Something that we are seeing
• 14:12
  consistent, really positive feedback about right now is a
• 14:17
  tool we introduced a few weeks ago called Dynamic Templates and
• 14:21
  a follow up to that was something we've called Remix.
• 14:24
  So Dynamic Templates have given users the ability to just enter
• 14:28
  free form field. Tell us exactly what you're
• 14:30
  trying to do. So like I am an SEO specialist.
• 14:36
  And I have some set of keywords that I'm trying to target and
• 14:39
  I'm trying to create Google ads. And then from that we will
• 14:43
  dynamically generate a template where we have an input field for
• 14:46
  the keywords that you mentioned you want to give us.
• 14:48
  And we have an input field for more context.
• 14:51
  And then we will output the ads that you've asked us to output.
• 14:54
  And that has been something that was kind of a big that, because
• 14:58
  it was something where like, yeah, it takes a lot of upfront
• 15:01
  work, but people pretty consistently have loved it and
• 15:05
  have just found it to be really powerful.
• 15:07
  So that is something where we are taking that power and trying
• 15:11
  to implement that elsewhere in the app.
• 15:14
  Interesting. So, Dina, welcome back I think.
• 15:18
  Maybe. And so maybe we can go to you
• 15:20
  for a second. And I I think I know one thing
• 15:23
  that we discussed a little. Bit over e-mail before this call
• 15:25
  was. That prompting was.
• 15:27
  Something that you guys thought a lot.
• 15:28
  About at Firefly and how do you help users know how to craft?
• 15:31
  Prompts create prompts. How do you maybe avoid the need
• 15:34
  for them to do prompt engineering?
• 15:35
  Or. Maybe you'd love to talk a
• 15:36
  little bit how you thought. About that at Firefly.
• 15:38
  And I think it relates. To this question of interfaces,
• 15:40
  how often do you even want to expose a a prompt at all to the
• 15:43
  user? Yeah, yeah, I think being able
• 15:47
  to the prompt is right now, right, because we do text to
• 15:50
  image. That's the main thing that you
• 15:52
  you use to generate the image of your that you're imagining.
• 15:57
  So most people, I mean this is a pretty new technology like
• 16:00
  prompting is basically the new Googling, right.
• 16:02
  Most people might not know how to prompt it properly.
• 16:05
  So if you look at some of the data, we're seeing like an
• 16:07
  expert, right, somebody who has spent a lot of time perfecting
• 16:11
  their prompt engineering skills going to get so much better
• 16:14
  results. Like 90 percentile results
• 16:16
  versus somebody who just says give me a picture of a dog,
• 16:19
  right. You have to add much more
• 16:21
  details on top of that. And I know we've been saying
• 16:24
  that prompt engineering in the future might go away.
• 16:27
  You know it'll be much, much, much easier to do prompt
• 16:30
  engineering. But from what I'm seeing is that
• 16:31
  you really do need to put in a good prompt to get a good
• 16:35
  result. So I think there are many ways
• 16:37
  to do it is you could out of fuel a prompt.
• 16:40
  So when you when you get a prompt, you can essentially use
• 16:43
  IML there to essentially. Predict what people want, and
• 16:47
  you might have your own data to essentially have an idea.
• 16:50
  So instead of typing the whole thing and having a whole
• 16:53
  sentence prompts, you actually help the user.
• 16:56
  And then I think what we do on the Fire Firefly website is that
• 16:59
  after prompting, I think it's really good to have predefined
• 17:03
  styles. So we have predefined styles as
• 17:05
  essentially in a form way. So you select a resolution and
• 17:08
  you select is it going to be classical style?
• 17:11
  Do you want to animate style? Right.
• 17:13
  Maybe that's something you don't want to put into your prompt,
• 17:15
  because it's kind of like. Um takes up your space right?
• 17:18
  So if you look at the journey prompts, they're very long,
• 17:21
  they're like photorealistic 3D extreme resolution and you keep
• 17:27
  adding V5V things. So.
• 17:29
  So I think that eventually will go away and this is something
• 17:31
  we're still working on. It's like how do you make it
• 17:34
  very easy, Just put what you really want in the prompt and
• 17:37
  then maybe have some type of canvas, predefined styles, drop,
• 17:41
  select, right combination of all these things to really help user
• 17:46
  to generate what they. Actually want and then I also
• 17:49
  really like generating variations.
• 17:51
  So then you learn from these things and you can map it back
• 17:54
  and you make your prompt suggestion better.
• 17:57
  So I think we're still at the very early stage where I'm not
• 18:00
  sure what would be the right combination of tools.
• 18:02
  But I think as we keep learning, as we keep collecting these
• 18:06
  feedback, we'll we'll have a better idea on how to really
• 18:10
  empower, you know, users to generate what they want.
• 18:15
  Artem, I know you mentioned prompting earlier and both you
• 18:19
  also. Talked about like I suggestions
• 18:20
  for prompts. I don't know if you guys also do
• 18:22
  like template. Jize prompt configuration
• 18:24
  similar to what Dennis mentioning or some of the.
• 18:26
  Other things like that. I'm curious, any insights you've
• 18:29
  had on how to abstract or handle prompting?
• 18:31
  From the user's. Perspective on your end,
• 18:32
  especially because you have a very garden, is very complex.
• 18:35
  Task I can create. These are arbitrary automation,
• 18:38
  so in some ways it's a more complex design space from a
• 18:41
  prompting angle. Yeah, I think the the the
• 18:46
  context is important here. So for instance for the
• 18:50
  creating, designing automations we want to be, we want to have
• 18:54
  as few restrictions as possible and then just let the user tell
• 18:59
  whatever they want. And like some users are very
• 19:01
  chatty and some users are very concise and so we kind of
• 19:05
  compensate by just trying to do as much of heavy lifting as
• 19:09
  possible on our end and just you know.
• 19:13
  Because again the surface area to cover there is so big that
• 19:16
  it's like very hard to come up with something template now when
• 19:20
  it comes to kind of other aspect in the product.
• 19:23
  So again we're an automation platform and we but if you want
• 19:26
  to say like, OK, I want to create some let's say outreach
• 19:31
  workflows for myself. So the task here is much more
• 19:34
  defined here. I want to generate tax and I
• 19:36
  think what Dina said, you know makes a lot of sense.
• 19:41
  It's in our context as well where you basically.
• 19:44
  There's a spectrum like 1. Spectrum is like you just it's a
• 19:47
  box where you have to put your entire prompt yourself that the
• 19:51
  end of the spectrum is just one button.
• 19:54
  It's just like generate outrage e-mail for me.
• 19:57
  And I think what we have in the middle is that sort of a
• 20:01
  template or the kind of the drop down selection where we guide
• 20:05
  the user through like OK, you're trying to create an Outlook
• 20:08
  Gmail. Now that we know the task, we
• 20:10
  know the dimensions across which we need specialisation or we
• 20:14
  need your. Input.
• 20:15
  And so we let them peak those dimensions.
• 20:18
  But then what we have at the end, we have actually we have
• 20:20
  researchers who that we're basically constantly evaluating
• 20:25
  a lot of different prompts, different combinations, how to
• 20:29
  like call out the model, how sensitive is the model to
• 20:33
  different changes that we make. And we do like you know formal
• 20:36
  evaluation. So we try to get a few kind of
• 20:41
  input points from the user but then what happens in the back
• 20:44
  end is basically construct the prop like construct the actual
• 20:49
  prompt that we want to send on the fly.
• 20:52
  So yeah I mean you know TLDR just thinking about you Access
• 20:57
  at large is you wouldn't fly a fighter jet with like a text
• 21:02
  message right. Like you need like the interface
• 21:05
  there is very kind of detailed overwhelming for someone who's
• 21:08
  not a specialist and but but I mean at the same time you don't
• 21:14
  want to put like a fighter jet like you know like really really
• 21:19
  complex UI in front of in front of the the the person.
• 21:23
  So I think what we're trying to do is like as quickly as
• 21:27
  possible to try to figure out intent and then once we know the
• 21:31
  intent just have a kind of highly specialised, highly kind
• 21:36
  of tuned something that would be like put user into a familiar
• 21:40
  place like because like users know how to click through with
• 21:43
  Wizards. Like for 20 years they have been
• 21:47
  like primed to go through this like very classic UX.
• 21:51
  And then I think you know the secret is that there is no magic
• 21:56
  bullet. You have to kind of think about
• 21:58
  your domain and make something that is familiar to the user,
• 22:02
  but at the same time kind of leverages all the, you know,
• 22:06
  kind of magic behind the scenes. That makes a lot of sense.
• 22:09
  So maybe let's move on to the topic of accuracy.
• 22:12
  And I think we all know that hallucinations are big.
• 22:14
  Issues with large language models and it's also hard to set
• 22:17
  user. Expectations around when will
• 22:19
  this be right and when might it be wrong and you need to check
• 22:21
  it. So maybe, Misty, we can start
• 22:23
  with you. Would love to get a sense of how
• 22:26
  you guys think about constraining the output of these
• 22:28
  models, making sure they're accurate, or helping users
• 22:30
  understand when it might be inaccurate and they're going.
• 22:33
  To yeah. So that is something that again
• 22:37
  we're going to continue to improve on.
• 22:39
  That right now the best way that we're ensuring accuracy with
• 22:42
  Jasper is with this toggle that we have in Jasper chat you can
• 22:46
  actually enable Google search. So that is something that we're
• 22:50
  seeing a lot of users use when they're not only wanting to
• 22:54
  creatively write content and generate content, but they're
• 22:56
  also wanting to have that fact checked against some resources
• 23:00
  from the web. And we will even including some
• 23:03
  prompts or we'll see users including some prompts to
• 23:05
  include URL so the references and those URLs.
• 23:09
  Well from what I've seen, for the most part they are real
• 23:12
  URLs, but there are still cases of those hallucinations where
• 23:15
  it's like factcheck.com and and it's not sorry that probably
• 23:20
  isn't really original, it's not go to that but it will be
• 23:23
  something made-up and so we are still recommending that people
• 23:29
  Fact Check anything that is stated.
• 23:31
  I mean, we've even had like just as a workshop, we've done like,
• 23:36
  right, the legal defence of robot marriage and.
• 23:40
  Like Jasper did an amazing job reading a legal defence and
• 23:43
  defending with all these historical Supreme Court cases
• 23:46
  that obviously never happened. So yeah, it will be something
• 23:51
  we're we're continuing to to add potentially like accuracy scores
• 23:56
  in the UI. That's not something that we
• 23:58
  have today, but but the capability is growing to be able
• 24:02
  to score accuracy and display that to a user.
• 24:04
  So something we'll be thinking about and implementing in the
• 24:08
  future. Maybe Dina, would love to get
• 24:11
  your take on the same space. I mean, I think you have a more
• 24:14
  creative. Products, Maybe there's less
• 24:15
  issues about fact. Checking per se at the same
• 24:18
  time, I assume you still may. Have issues with.
• 24:20
  The model being probabilistic and maybe not doing exactly what
• 24:23
  I. Expect as a user.
• 24:24
  So have you guys had to deal with problems like that and how
• 24:27
  have you approached it? Firefly.
• 24:28
  Yeah, definitely that's it's a big problem, especially when
• 24:31
  you're drowning images. It's, it's about the, like we do
• 24:35
  have on the web, the web UI, like thumbs up, thumbs down.
• 24:37
  And we do ask a lot of feedback. So first is like, is that what
• 24:41
  you really actually wanted, right, That that's the
• 24:43
  information we want. And then the accuracy, is this
• 24:45
  quality good? Because it's possibility, right?
• 24:47
  We'll generate something that's poor quality, that's, you know,
• 24:50
  corrupted also. It's also content moderation.
• 24:53
  I think that's a big part and it's FW.
• 24:55
  We want to make sure we're we're keeping our users safe and not
• 24:59
  accidentally generating something that is not safe.
• 25:02
  And so we let them kind of report these issues and we save
• 25:05
  that and bizarre like when we're developing our foundation model,
• 25:08
  they get really relies on the data like the causes of all
• 25:11
  these inaccuracies like goes back to the data And so we have
• 25:15
  to improve our model based on these results or also improve
• 25:18
  based on the prompt side, based on the, you know like the post
• 25:21
  generation side. So there's a lot of things we do
• 25:25
  to essentially interact currency that it's safe and I think it's
• 25:29
  going to be a big field because frankly right like we have to
• 25:33
  now do it yourself essentially. So, so very excited for you know
• 25:38
  different people to work on these and this of W Philtres
• 25:41
  classifiers, models, Jen AI content moderation type tools to
• 25:45
  sort of make that process easier because that's where the kind of
• 25:48
  like that's the last mile type problem that becomes really
• 25:52
  relevant when you go to production.
• 25:54
  Hmm. And Arta maybe.
• 25:56
  Would love to get your take on maybe two of the things you
• 25:58
  mentioned. So one is.
• 25:59
  This trust and safety layer, is that an?
• 26:00
  Issue for you guys that you. Had to think about.
• 26:02
  And then #2I. Think Dina brought up the idea.
• 26:05
  Of measuring performance and getting feedback and using that
• 26:08
  to improve the model over time. How do?
• 26:10
  You guys think about what signals to collect and how to.
• 26:13
  Quantify performance over time. Just throwing in a quick
• 26:16
  two-minute warning. Thank you.
• 26:18
  Yeah, I mean for sure it's an issue because like you would be
• 26:21
  surprised at things that people want to automate, you know you
• 26:30
  know in terms of accuracy. I think we kind of we knew it
• 26:34
  was going to be a challenge so we kind of went overboard.
• 26:37
  So we basically designed a language DSL, a domain specific
• 26:41
  language for defining automation that has been built, properties
• 26:45
  that make it easy for us to verify the accuracy and then we
• 26:50
  also again for us it's we're not trying to take humans out of the
• 26:53
  loop. So for us is the automation is
• 26:56
  like we want to show that we want to like human is the people
• 26:59
  are the best kind of verifiers of like accuracy.
• 27:02
  And so we show it to them and say like, hey, is this your
• 27:06
  intent? And then along the way like
• 27:08
  dinner was mentioning, we collect a lot of telemetry data
• 27:11
  like thumbs up, thumbs down, How many attempts does it take them
• 27:15
  to get to the automation they want?
• 27:17
  Are they happy with it? How often do they use it
• 27:20
  afterwards? And then of course all this data
• 27:22
  you're feeding back to our kind of, you know, creation cycle.
• 27:26
  So we continuously keep improving that.
• 27:30
  Got it. That's interesting.
• 27:32
  And let's see, we may have one minute left, maybe we can do a
• 27:35
  lightning round. And one last question, I do
• 27:37
  financially in the UK side, latency and performance.
• 27:39
  Can be a huge. Deal for LM.
• 27:41
  Applications that massively affect UX and iteration speed of
• 27:45
  the user maybe really quickly. Would love to talk to each of
• 27:47
  you guys, maybe starting. With Misty, then dinner, then
• 27:49
  Artem on has been an issue for you and have you done?
• 27:52
  Anything interesting technically to improve latency?
• 27:55
  Latency and errors spikes have been an issue, so more context.
• 28:00
  Casper is an application layer that sits on top of these
• 28:03
  models. And so a lot of the time that
• 28:05
  latency and those error rates are sort of out of our control.
• 28:09
  And we have built data monitoring and reporting using
• 28:12
  data docs. So we have really good insight
• 28:15
  into any time those spikes are happening, where wait times are
• 28:18
  becoming especially long or or air rights are becoming
• 28:22
  especially high. We're then able to post messages
• 28:25
  and the dashboard. If there's something especially
• 28:27
  problematic, like hey, Jasper's running into some snacks, Please
• 28:30
  be patient. We're working on it and then we
• 28:33
  are also like. Probably more importantly we
• 28:36
  have implemented fall back strategies anywhere where that's
• 28:39
  been identified as an issue. So we're able to quickly switch
• 28:43
  over to a different model or different provider in order to
• 28:47
  hopefully deliver better performance and get people
• 28:49
  responses as often as possible and at the highest quality
• 28:53
  possible. Yeah I would say we do with that
• 28:56
  everyday scaling. It's honestly a good problem to
• 28:59
  have when you're when you're expectation was exceeded by so
• 29:03
  much that now you have to scale and change computer like stop
• 29:07
  something to send compute there. So definitely a problem buddy I
• 29:12
  I think I'm excited to see especially on the serving side
• 29:15
  how do you serve but I think if you look at our website our
• 29:18
  latency is pretty good because we've been serving these ML
• 29:21
  based features on top of Photoshop and different
• 29:23
  products. So building on previous
• 29:25
  generation of of architecture but also like exploring new
• 29:29
  things to make it better kind of there are some things where we
• 29:34
  can just use traditional boring solution to the latency which is
• 29:37
  caching and and so we we have we do that.
• 29:41
  So if like you wanted to create an automation it's going to be
• 29:45
  and you described it the same words as I, I don't need to run
• 29:48
  through LM to get you. So we do that and then I think
• 29:54
  I'm super excited about small models because then we will be
• 29:57
  able to run them on the problem which is like highly
• 30:00
  specialised, much smaller models that that we can distribute.
• 30:04
  So are you. Are you running any models in
• 30:07
  the browser today or not yet? I know you're home extension.
• 30:09
  Yes, we are. We are.
• 30:10
  We have a very intensive like, it's amazingly, it's so good.
• 30:14
  But like you don't. You don't have to pay for it, so
• 30:19
  yeah. Awesome.
• 30:20
  I I think we're probably a little overtime.
• 30:23
  Don't know if the moderator needs.
• 30:24
  To come on, but. Thank you all so much for.
• 30:26
  Doing this, Yeah. Thanks for inviting me.
• 30:32
  This is awesome. Thank you so much, Davis, Misty
• 30:35
  Ardham, Dina, especially Davis for guiding the conversation.
• 30:38
  It's not an easy feat with, like a couple of people.
• 30:40
  So that was awesome. Definitely.
• 30:43
  Yeah. All these talks will be
• 30:45
  recorded. And then we're also, I guess
• 30:47
  there were no slides for this one, so I will not be sharing
• 30:50
  slides. But thank you all so much for
• 30:52
  joining us. This was awesome.
• 30:54
  All right. So long.
• 30:57
  Yeah. Hi, this is Michael Eric.
• 31:01
  I'm the founder of Pametan Data Innovation and Confetti AI and
• 31:05
  this is a quick public service announcement.
• 31:07
  If you're tired of having nightmares about the machine
• 31:10
  learning models you have in production, then I encourage you
• 31:12
  to subscribe to the ML OPS Community Podcast.
• 31:15
  It's where all the. Cool ML OPS kids hang out and
• 31:18
  learn how to tame their ML models.