Latest / Agentic Conversations (formally mlops.community) / Using Large Language Models at AngelList // Thibaut Labarre // #171

Transcript

Using Large Language Models at AngelList // Thibaut Labarre // #171 · Agentic Conversations (formally mlops.community)

• 0:00
  My name's Tibo. I'm an engineering lead at
• 0:02
  AngelList and I don't drink coffee.
• 0:05
  I drink liquid death exclusively.
• 0:08
  This is my go to drink so. Hello and welcome to another ML
• 0:13
  OPS Community podcast. I am your host Dimitrios, and
• 0:15
  today I'm joined by none other than Mr. Ryan Russell.
• 0:20
  How you doing, dude? Doing all right.
• 0:22
  It's been a been a minute, but yeah, glad to be here.
• 0:25
  You graduated from guest to co-host and now?
• 0:31
  You were such a good Co host the first time we had you on.
• 0:33
  I had to have you back and I am very happy that I did.
• 0:37
  We explored just about everything when it comes to
• 0:41
  using Open AI. Models, building them into your
• 0:46
  product with. To go in this conversation, did
• 0:50
  you have any main takeaways that you want to share with us?
• 0:54
  I think they're similar to what you said but I think a lot of
• 0:57
  it, you know I'm an infrastructure person in general
• 1:00
  there's over engineering crap is my job.
• 1:01
  No, I I I was. I'm always intrigued to hear how
• 1:05
  today again with the hype around these LM models and especially
• 1:09
  as like a as a service being provided by so many different
• 1:12
  companies such as Open AI, how people are building around them
• 1:16
  and how they're extracting value from them because it's I think,
• 1:20
  right. The value is more than just
• 1:21
  wrapping it up in vision software.
• 1:24
  There's a lot that goes into it. Yeah, It was interesting how
• 1:26
  Tebow went through all that and talked about those challenges
• 1:31
  and the things that he had to do to make specifically relay his
• 1:35
  product for Angel list. Kind of a viable solution and
• 1:38
  really bring value out of that. Yeah, so he breaks down relay
• 1:43
  what it is. They just released it today on
• 1:45
  the day of this recording, which by the time this comes out will
• 1:48
  be like a month ago. But.
• 1:51
  I was fascinated with. His dedication to figuring out
• 1:57
  if. A feature.
• 2:00
  Is actually viable? And if it's worth pursuing?
• 2:05
  And not getting caught up in the over engineering like you were
• 2:10
  saying and so him going from zero to 1 on the two different
• 2:15
  use cases that he talked about I thought were awesome to see.
• 2:19
  And every time we asked them all these things about well what
• 2:23
  about if do you want to use a new model maybe you want to try
• 2:26
  a different API's like anthropic or cohere.
• 2:30
  He's like, yeah, but we've got so many different features that
• 2:34
  we want to in. We want to create.
• 2:37
  Why would I waste my time trying to figure out a new model when I
• 2:42
  can spend my time? Creating new features with the
• 2:46
  model that I already know works quite well.
• 2:49
  And I mean, I, I we questioned with it, right, challenged a bit
• 2:52
  on the, the scale, right, like what is the upper limit And
• 2:54
  there's definitely risk when you tie yourself to a third party
• 2:58
  vendor, there's risk when you tie yourself to a managed cloud
• 3:00
  service as well, right. I loved hearing how he was
• 3:03
  mitigating those risks, how they thought through.
• 3:05
  It wasn't just like, Oh yeah, I didn't really think about that.
• 3:07
  I was like, no, we thought through this process.
• 3:09
  We thought about it beforehand. We made a deliberate decision to
• 3:12
  proceed the way we did and I appreciated that.
• 3:16
  Again, that's that like narrow focus on this is what we're
• 3:19
  trying to deliver. Here's the value we want to add.
• 3:21
  We're not worried about this limit right now, but we'll get
• 3:25
  there. Maybe we'll worry about it when
• 3:26
  it's a little bit more pressing. And I think it's brilliant that
• 3:30
  they're just sticking to their guns and trying to.
• 3:34
  Tackle the low hanging fruit before they.
• 3:37
  Add complexity to the scenario and as he was saying, yeah,
• 3:42
  we're going to look at fine tuning or creating, potentially
• 3:45
  creating our own large language model and going after all of
• 3:49
  that. But first, let's just tackle
• 3:52
  these use cases that we know we can use Open AI for and we know
• 3:56
  it brings a huge value add to the end user.
• 3:59
  So hats off to him for that. I think we should just go ahead
• 4:03
  and get right into the conversation.
• 4:05
  But before we do, I want to mention to everyone.
• 4:08
  It would be a huge help if you have not already hit, follow or
• 4:13
  subscribe or like or rated. This podcast means the world to
• 4:18
  us if you do. And also that one thing that
• 4:22
  would help us the most would be if you share this podcast with
• 4:25
  one friend so that we continue to spread the good word of the
• 4:31
  ML OPS community. And without further ado,
• 4:48
  That you're a huge fan of Isaac. What brings that about Isaac
• 4:55
  Asimov? Like some of the best books I've
• 4:58
  read as a kid, actually, really early on.
• 5:00
  And UNCLE gave me foundation and I just devoured the series, like
• 5:04
  all the prequels, all the sequels.
• 5:05
  So yeah, really big fan. And I think his vision is still
• 5:09
  relevant today, especially as we see all the advances in AI
• 5:13
  machine learning. Did you watch the series?
• 5:15
  Yep. On Apple TV.
• 5:17
  Yeah. I think you liked it.
• 5:19
  Yeah, yeah, it's different from the book.
• 5:22
  But honestly, like, you know, for me, like going back into
• 5:24
  this universe is just just just amazing.
• 5:27
  It's like a dream come true. What captured your imagination
• 5:30
  the most? I think the portrayal of Leon,
• 5:33
  like the, you know, the Emperor, and like the way they render
• 5:36
  like the three generations living together.
• 5:38
  I think that's always like, interesting.
• 5:40
  Like, really well done. Yeah.
• 5:42
  And so for those who don't know maybe what it what is it that we
• 5:47
  are talking about right now? The Foundation series, this was
• 5:49
  a book that was published in the 80s, right?
• 5:51
  Or was it even? Yeah, maybe if you're in earlier
• 5:54
  than that, I think little earlier, you need to ask.
• 5:57
  Yeah, like some. Very sci-fi book, but it
• 6:02
  explores a lot of these existential questions.
• 6:04
  And yeah, Asimov is the one who created the Three Laws of
• 6:08
  Robotics, right? The robots should obey orders
• 6:13
  that they should not harm any human.
• 6:17
  And you know what I should have? Probably you'd find.
• 6:22
  That was kind of the confusing part.
• 6:24
  Is that? Instantly.
• 6:28
  It's not clear like the whole thing about these books that is
• 6:32
  so brilliant is how he explores the grey area.
• 6:35
  That's right. Not my.
• 6:37
  My biggest takeaway is that it's yeah, you can have these three
• 6:41
  laws. But at the end of the day,
• 6:44
  there's a lot of room for interpretation, and especially
• 6:47
  depending on the events and depending on how reality plays
• 6:51
  itself out, you can get yourself into sticky situations.
• 6:56
  That's right. It's also the question of like
• 6:58
  saving one person, you know by maybe harming another one.
• 7:02
  Like with Call should the robot make?
• 7:04
  And especially as it scaled to all of humanity.
• 7:06
  Right, that was the third law, right?
• 7:07
  You have robots got to protect some existence so long as it
• 7:11
  doesn't violate the first two laws, right?
• 7:12
  That's. Perfect.
• 7:13
  Well, awesome. I didn't think we were going to
• 7:15
  take that turn, but let's let's get into this, man.
• 7:19
  So you're working at Angel List right now.
• 7:21
  Y'all are doing some pretty wild stuff when it comes to AI and
• 7:25
  ML. It's probably worth just giving
• 7:28
  us a bit of background on on what the ML journey has been
• 7:33
  like and then how it's changed since the emergence of ChatGPT
• 7:39
  and all that fun stuff. Yeah, for sure.
• 7:41
  So I joined a list about like three years ago, and this was
• 7:45
  for me a dream job. Like Angel invested on the
• 7:48
  platform for a few years. And yeah, getting to work on my
• 7:53
  hobby, it was a lot of fun. And my background was working on
• 7:59
  natural language processing at Amazon, basically parsing all
• 8:02
  customer feedback, right, so that we could package it for PMS
• 8:06
  throughout the company to to use to improve their products or
• 8:09
  improve experiences. And so naturally I gravitated
• 8:12
  towards problems that allowed me to, you know, put some machine
• 8:16
  learning in place at Andrew Lists.
• 8:18
  And so one of the early problems was the ability to classify news
• 8:22
  articles so that we could actually route them to the
• 8:25
  investor dashboards for any companies that they've invested
• 8:28
  in. And so as you can imagine, I
• 8:30
  built a simple classifier, label the bunch of articles and had
• 8:34
  this running pretty smoothly to media a month or two to deploy
• 8:37
  and get out the door. And so that was like step one,
• 8:40
  Like, you know, pretty basic, run-of-the-mill machine learning
• 8:44
  use cases. Another thing is, for my own
• 8:48
  coding work, this was the early days for GitHub Copilot.
• 8:52
  Like they had just launched, I think you know, an open beta
• 8:55
  that was free to use. And it was maybe one of the
• 8:59
  engineers on the team trying it out and having fun with the
• 9:02
  outputs and realising that it actually helped me write unit
• 9:05
  tests a lot faster than I could before.
• 9:07
  So it was the early days, but yeah, it was.
• 9:10
  You could see glimpses of where we are today.
• 9:13
  You glossed over something there that I think is interesting, and
• 9:17
  you said, yeah, it took like a month or two to deploy when it
• 9:20
  came to the. Your first project, the hit.
• 9:24
  Were you walking into an already mature organisation?
• 9:27
  Break down what it looks like at Angel List.
• 9:31
  Yeah, for machine learning, right.
• 9:33
  Yeah. So I was probably maybe the like
• 9:36
  the 1st engineer on the team to have some experience working
• 9:40
  with machine learning. And so yeah, it was I think not
• 9:43
  a mature organisation from that perspective at all.
• 9:46
  Like there were no data scientists.
• 9:47
  Like there were no, you know research scientists.
• 9:49
  Like no one focused on machine learning as a core competency.
• 9:53
  And I think the team had explored use cases, maybe off
• 9:57
  the shelf APIs that you could call for for some use cases that
• 10:01
  under the hood used machine learning models, but nothing
• 10:05
  that had been trained specifically for annual list use
• 10:07
  cases. So yeah, it was pretty new.
• 10:09
  What's great about this company is it's like willing to try
• 10:12
  things out and really gives a lot of autonomy to all employees
• 10:16
  on the team, right. Like when we join, we're told
• 10:18
  hey, you're the, you're the founder for your one person
• 10:20
  startup, right? And we trust you.
• 10:22
  We hire high judgement people and like you go to you're saying
• 10:25
  you don't need to, you know ask for too many permissions to try
• 10:28
  things out. And so I think that that clicked
• 10:30
  and that allowed me to experiment, get things out the
• 10:33
  door and so two months, I mean in retrospect it's maybe a bit
• 10:37
  long, but I think it's a lot shorter than what it could have
• 10:40
  been at another organisation and I had to go through you know, a
• 10:44
  lot of approval processes and you know getting sign off like
• 10:48
  before starting the project. It's interesting always to hear
• 10:50
  when someone comes into a company and like data science is
• 10:54
  not, is not existent. I've been there, I've been, I
• 10:56
  feel like, you know, I've been in companies where it's just
• 10:57
  kind of like it's gorilla data science, right.
• 10:59
  You're just you're the only person who might have an idea of
• 11:01
  what's how to do something. And then you're it's up to you.
• 11:04
  Also build whatever you're going to build.
• 11:07
  You have to now manage sort of a platform, right for this thing
• 11:09
  you have to be you're now the owner of this.
• 11:11
  So how does that look like for you going into again a company,
• 11:17
  not very much data science. You don't have machine learning
• 11:20
  engineers. How is that now?
• 11:21
  How is that when it got there? Like what is that that journey
• 11:23
  look like to actually building out something that can be useful
• 11:26
  in an AI sense? I had a bit of a head start
• 11:28
  because there was very familiar with the AW services that
• 11:32
  supported you know, machine learning and text specification.
• 11:35
  I used Comprehend, I trained the custom model there and so a lot
• 11:39
  of the infrastructure was ready for me to apply to the problem.
• 11:44
  I also picked a textbook problem so like the complexity was
• 11:49
  pretty low. We're talking news article
• 11:50
  classification, so that was also maybe a bit of a a hack to get
• 11:54
  to a working solution faster. But yeah, eventually end up
• 11:59
  maintaining this over its life cycle and like onboarding more
• 12:02
  engineers to work in the in in the same framework to apply
• 12:06
  these technologies to other problems.
• 12:07
  And so I think we've really built a lot of competency around
• 12:11
  this but more from a practitioners perspective like
• 12:14
  we haven't spent too much time fine tuning what was available.
• 12:18
  We really have leveraged off the shelf services.
• 12:21
  So I think that made the maintenance and operations of
• 12:25
  the systems a lot easier than than otherwise.
• 12:29
  And do you feel like it cause? That's something I think those
• 12:33
  trade-offs are things that people think about a lot, where
• 12:38
  it's like, yeah, we can just get up and running, leapfrog,
• 12:42
  basically go from zero to one really quick.
• 12:45
  That it. But then you also are giving up
• 12:48
  a lot, right? That's the trade off is that you
• 12:51
  don't get to necessarily be very opinionated because everything
• 12:54
  that you're using makes those choices for you.
• 12:57
  And in your experience, it's been good enough to just go with
• 13:02
  everything off the shelf and take it and figure out how you
• 13:05
  can work within those constraints.
• 13:07
  Honestly, it was not good enough.
• 13:09
  I mean it was good enough for a period of time but there were
• 13:12
  like core limitations like things that just didn't you know
• 13:16
  like work for us and like 2 examples like 1 comprehend when
• 13:20
  you deploy your model like you have to scale it to I think the
• 13:24
  lowest priced point custom model that you can deploy is like
• 13:28
  $1000 per month. And basically they give you one
• 13:32
  server and you can scale it up or down like there's no server
• 13:35
  less solution to that and $1000 worth, pretty expensive for what
• 13:38
  we were doing. And so that wasn't great.
• 13:42
  And the second thing was we needed to go a little bit deeper
• 13:46
  into the news articles themselves.
• 13:47
  We needed to understand, for example, if there was an
• 13:49
  acquisition, who is the acquirer, who was the inquiry,
• 13:52
  right. And that does not come as like a
• 13:54
  default text classifier off the shelf.
• 13:57
  And so there were limitations and we actually were able to
• 14:01
  kind of deprecate the whole thing and rewrite the whole
• 14:04
  system in a day. And I'm not kidding like in one
• 14:07
  day by leveraging large language models like Open AI.
• 14:11
  So basically a prompt, no fine tuning, just asking to classify
• 14:15
  those articles according to the categories defined.
• 14:18
  Maybe one or two. You know, like future arts
• 14:20
  examples like basically like replaced and dramatically
• 14:25
  simplified like what I had spent two months building and allowed
• 14:29
  us to go even deeper to do things like ohh, now we want to
• 14:31
  know who the acquirer is, who the acquirer is.
• 14:34
  We also don't care about raises from VC funds.
• 14:37
  We only care about when the company raises.
• 14:39
  And all of that was super straightforward to write as
• 14:41
  just, you know, prompt improvements.
• 14:44
  And then that was with GPT 3, right?
• 14:47
  And when, you know, 3.5 came along and then GPT four came
• 14:51
  along, like burst system just got better, you know, with us
• 14:55
  and doing 0 work and it was cheaper also because now we
• 14:58
  weren't paying for that one server to keep running all the
• 15:01
  time, we were just paying per per prompt, per request.
• 15:04
  And so, yeah, that's just how the world has shifted a little
• 15:08
  bit and how now we are actually empowering a lot more people to
• 15:13
  basically train similar models for similar use cases, right,
• 15:16
  without with a way lighter infrastructure actually to do
• 15:19
  this work. So yeah, you went from cloud
• 15:22
  managed. Write a WS's solution to.
• 15:25
  Now you're just purely vendor third party option, right?
• 15:28
  Essentially, I'm sure there's a still in there, but you've now
• 15:31
  just built your system around open AIS chat model pretty much.
• 15:36
  Yeah, that's right. OK.
• 15:39
  And I am assuming that you're having to do something.
• 15:44
  Because Open AI doesn't know about the most up-to-date
• 15:49
  acquisitions or funding rounds, et cetera, et cetera.
• 15:52
  So are you doing some kind of? Retrieval augmented generations
• 15:58
  or how does that look? So you basically feed the news
• 16:02
  article in the context, right? So it it is provided with like
• 16:05
  the recent current data. And like Arminius case is really
• 16:09
  data extraction. Which means that we're just
• 16:12
  asking you, hey, given this news article, what are the key pieces
• 16:16
  of information? What are the named entities?
• 16:18
  Who's the acquirer? Who's the inquiry?
• 16:20
  Yeah, open the iBooks. Like, awesome for these kinds of
• 16:23
  use cases. And we can also double check
• 16:25
  that whatever we've extracted is actually in the source, right?
• 16:27
  So like preventing hallucination is easy with use cases like this
• 16:31
  one. How are you doing that?
• 16:33
  I imagine you're not just like double checking human wise.
• 16:36
  We still have some human in the loop processes, especially for
• 16:40
  like news articles that we published to our sites.
• 16:43
  Like doing a little bit of duping like picking the best
• 16:45
  article is still something that we do manually, but over time we
• 16:49
  could also get rid of that once we're confident enough in our
• 16:53
  models. And like in practise they've
• 16:55
  performed like amazingly well, like for those data retrieval
• 16:59
  tasks, I think like GPT 4, is that be 99% accurate, right?
• 17:04
  Like, I mean it's like human accuracy, right?
• 17:06
  Because humans make mistakes as well.
• 17:08
  So right now we we trusted for actually most of the basic tasks
• 17:14
  in the process, right? And we start trusting it with
• 17:17
  higher and higher complexity tasks that we send it its way.
• 17:22
  Do so if I if I'm understanding this correctly, it's basically
• 17:25
  like you are having a programme scour the Internet for the most.
• 17:32
  Up to date news and then when you get something that at some
• 17:37
  point so how do you first classify it that it's relevant
• 17:42
  to what is happening on Angel list Just if it has a mention of
• 17:44
  a company pretty much we actually the problem is pretty
• 17:47
  basic is like is this relevant to a company raising money
• 17:51
  exiting or you know having a product launch.
• 17:54
  And then from there we ask if it's an exit, is it an
• 17:58
  acquisition? And if it's an acquisition,
• 17:59
  who's the acquirer and who's the acquirer?
• 18:01
  And that gives us what we want like structure.
• 18:05
  It has like nice like Jason blobs that we attach to our data
• 18:08
  model. I imagine you have had to.
• 18:12
  Fight for why this is actually valuable, like a value add to
• 18:17
  the Angelis product. I'm guessing that it.
• 18:21
  At the core, it's making the investors who are on Angel list.
• 18:26
  Be more informed. But have you thought about how
• 18:30
  to actually quantify that? Like do you have metrics around,
• 18:34
  hey, you know, these open API calls are costing us XML, but we
• 18:38
  see that the engagement has gone up this much, which can signify
• 18:42
  more investors spending on Angel list or something like that.
• 18:46
  So this functionality was actually something that we were
• 18:51
  paying people to do, like we actually were working with
• 18:54
  contractors to do some of the manual classification before,
• 18:57
  right? So I did not have to make the
• 19:00
  business case for the need for this feature.
• 19:02
  It's actually one of the top requested feature from
• 19:04
  investors, right? Obviously, you've put money, a
• 19:06
  lot of money into a company like you want to know how it's doing,
• 19:09
  you know what's going on with it.
• 19:11
  So we don't have a formal, I think, ROI analysis of how
• 19:16
  accurate classification of company news and assignments to
• 19:20
  investments gives us in terms of engagement on the site.
• 19:22
  But we know we need this feature, and we know that we are
• 19:26
  now able to do it way cheaper than we were able to do it
• 19:29
  before. Because, yeah, Open AI costs are
• 19:31
  a fraction of like the contractor costs that, you know,
• 19:35
  we were paying before. I was thinking about something.
• 19:39
  This is kind of a challenge question.
• 19:40
  You might want to cut this out later, but.
• 19:43
  Alright, that's a great way to start it, yeah.
• 19:46
  This is what I'm going to say. So there's there's a lot of talk
• 19:49
  around, you know, obviously there's a lot of hype around
• 19:51
  these. Open AI's solution?
• 19:54
  We've seen a lot of models, large language models, come out
• 19:57
  since and kind of resent themselves right to the public.
• 19:59
  There's a lot of, I think there's some discourse around
• 20:02
  how valuable it is to build applications that literally just
• 20:06
  kind of wrap what these vendors are providing and that the real
• 20:11
  value is actually like data, the data that's being used, right,
• 20:14
  that's like the actual value is. So I think going off of
• 20:16
  Demetrius question a little bit in terms of the ROI, the value
• 20:19
  add, right, you're spending X amount obviously on on open AIS,
• 20:23
  you know the API side of things provide this as a service.
• 20:27
  From a long term perspective, how do you think this product
• 20:31
  could evolve to be even more valuable or you know what is the
• 20:34
  next, what's the next value add if you can talk about that?
• 20:37
  Yeah, I mean it's a great question and this is definitely,
• 20:40
  these are areas that we're exploring.
• 20:41
  What I was describing with the news article classification use
• 20:44
  case was basically the first lightbulb moment of wow, this
• 20:47
  stuff we were doing manually. And with a lot of effort we can
• 20:51
  actually replace it with, you know, simpler and more
• 20:53
  straightforward solutions really quickly.
• 20:56
  And that led to. Where else can we apply this and
• 20:59
  what else can we do with this technology?
• 21:01
  And so the point about using a lot of the data and a lot of the
• 21:07
  information that's available to train custom models or basically
• 21:12
  on top of the base foundation models like train maybe focused
• 21:17
  layers that are dedicated to a subset of our data.
• 21:20
  For example, maybe the subset that's relevant to a single
• 21:23
  customer, so we can start having them interact with their
• 21:26
  information, let's say all your investments.
• 21:28
  For example, and start interrogating, interrogating it
• 21:31
  just like you would have a general conversation with Shaft,
• 21:33
  right? Like what if you could say what
• 21:35
  investments did I make in this industry, right, in the space
• 21:38
  industry that we're at the seed level and how have now raised
• 21:43
  the series be like that is a request that reasonably could be
• 21:47
  answered by a model that's trained on all the investments
• 21:49
  that you've made. And so, yeah, these are areas
• 21:52
  that we're exploring in like the part of the company that I'm
• 21:56
  working on, but it's obviously really caught on and like.
• 21:59
  Everyone's now thinking through how can we simplify, how can we
• 22:02
  automate, and how can we improve the experiences that we're
• 22:05
  building with these new tools, Right.
• 22:08
  It's become a building block, really.
• 22:10
  OK, I like that answer. Good.
• 22:11
  It seems like it's kind of this race now around there's a lot
• 22:13
  around the hype right around building any kind of experience,
• 22:18
  I'll call it, around large language models.
• 22:20
  And so I think a lot of companies right now want to be
• 22:24
  part of that, right. Do you think there's value in
• 22:27
  companies taking their own data and reading their own
• 22:30
  foundational models to build these types of solutions off of?
• 22:32
  Or do you think that it's better to just take something off the
• 22:35
  shelf as pre trained foundation model and tune it to the needs
• 22:39
  of whatever it is you're industry is or whatever your
• 22:43
  problem is? Yeah, the.
• 22:46
  And sort of from my perspective is that I think the return on
• 22:49
  investment is obvious and really straightforward with just taking
• 22:53
  off the shelf models and applying it to all those areas
• 22:56
  that require human input today, right, like from the most basic
• 23:00
  ones like and going up to the more and more complex ones over
• 23:02
  time, right. And so I think this is probably
• 23:05
  our biggest investment, like maybe the 70% bet right now is
• 23:08
  like, yes, let's focus on those. We know, we clearly know what
• 23:11
  you need to do. These are the workflows that
• 23:13
  we've operated right every day and like if we can reduce the
• 23:16
  costs over time. For this like we can accelerate
• 23:18
  the pace of you know, investments going into companies
• 23:21
  and increase the rate of innovation, right.
• 23:23
  That's really a primary goal. But we still want to reserve
• 23:26
  like 30% of our time to explore and see what else can be built,
• 23:30
  what are the new capabilities that can be unlocked with these
• 23:34
  technologies. And that's where those new
• 23:36
  experiences where we leverage the data that we already have
• 23:39
  like on behalf of our customers basically package it in a way
• 23:42
  that allows novel capabilities that they've never seen before.
• 23:45
  I think that's maybe the more exciting.
• 23:47
  The next step and obviously requires maybe a heavier lift,
• 23:52
  right like accessing the off the shelf APIs.
• 23:54
  One day you get a working news article classifier I think to
• 23:58
  train your own data. Like there are a few companies
• 24:01
  out there that are helping and building the tooling to allow
• 24:04
  you to do that right on top of the lemma models like the open
• 24:08
  models. So these are.
• 24:10
  Still on the prototype and, you know, hackathon stage?
• 24:13
  Yeah, for us. But we see a lot of potential
• 24:16
  there and we definitely want to, you know, see how far we can
• 24:18
  take. Those, the way that you're
• 24:20
  looking at it is let's figure out what's the easiest that we
• 24:24
  can do because we have this new. Incredibly powerful tool that we
• 24:30
  can use. And so everything that we can
• 24:34
  hit with this tool, let's go for it.
• 24:36
  And then we can graduate to the more complex scenarios and
• 24:40
  figure out how to go about that. But there's just so much surface
• 24:44
  area that we can cover with the. Tool that we have currently that
• 24:50
  we can do that and the other piece that I really like about
• 24:52
  what you're doing is you're not. Saying.
• 24:59
  AI is our product. It's you have a clear product
• 25:03
  already, you're just enhancing it with a I and what I've seen
• 25:06
  as the best user experiences with these different AI
• 25:11
  capabilities that have come out. It's not reimagining a new tool
• 25:16
  that has a I at the centre, it's I have my tool now, how can I
• 25:20
  augment my tool with AI? And it really feels like that's
• 25:24
  how you're looking at it. And I like that, what I also AM.
• 25:29
  Pretty curious about because you mentioned that the culture at
• 25:32
  Angel List is. People told you come in, you're
• 25:36
  like a one person startup. Go and get after it.
• 25:39
  You have to really be thinking about.
• 25:42
  How to create something from this?
• 25:45
  Mentality of alright, if I'm going to bring a new product
• 25:49
  into fruition, what am I going to create?
• 25:53
  How is it going to look and so? It feels like you need to.
• 26:00
  Be very strong on the product side too.
• 26:04
  How have you thought through that?
• 26:06
  Have you? Evolved in your way of justice
• 26:10
  being a machine learning engineer and then now thinking,
• 26:13
  OK, if I'm going to implement these new features, what do I
• 26:17
  need to think about? Yeah, it's a great question.
• 26:19
  So we work as obviously work as a team, right.
• 26:22
  So you're maybe the founder for your one person startup, but
• 26:25
  it's OK to work with cofounder and you can make it a two person
• 26:30
  startup. And so we have like amazing
• 26:32
  product managers that are really helping shape those visions and
• 26:37
  flag like. You know, bake like the most
• 26:41
  important releases that we can build for, for the company or
• 26:45
  for customers. I'd say also it's expected that
• 26:48
  as an engineer we come in as product engineers first and
• 26:51
  foremost, right? And you're expected to wear a
• 26:53
  lot of hats, like you're expected to be able to go
• 26:56
  through the full stack, be able to build as like the front end
• 26:59
  as well as the back end, like for your your feature, right?
• 27:02
  Like the entire slice of the product.
• 27:04
  And you're also expected to be able to go and talk to the
• 27:08
  different. The user is either external or
• 27:11
  internal for what you're building and shipping.
• 27:14
  And so, yeah, I think a culture that really emphasises that
• 27:17
  ultimately, like we we like to say that because promotion at
• 27:21
  Angel List is when you leave the company to start your own
• 27:23
  company, right? And so a lot of people who come
• 27:25
  in are either past founders or people who want to become
• 27:28
  founders. And so any experience they can
• 27:30
  get, you know, doing everything is, you know, experience that
• 27:34
  people find valuable. Everyone.
• 27:38
  My name is Aparna, founder of Verizon, and the best way to
• 27:40
  stay up to date with ML OPS is by subscribing to this podcast.
• 27:46
  And so then there is around on Angel list.
• 27:49
  They raise their seed round. Yeah.
• 27:53
  Talk with the products. Yeah, yeah, It's so cool
• 27:56
  thinking about that whole life cycle.
• 27:59
  You all just put out a new tool recently, right?
• 28:02
  Today I think. And today we'll timestamp
• 28:05
  because this probably won't go out for another month.
• 28:08
  So July 20th, tell us about what you put out and what the process
• 28:13
  was of like creating that. Yeah.
• 28:16
  So I talk about new particle classification as the first use
• 28:19
  case. And we quickly found that we
• 28:21
  could apply the same technology to the investment documents that
• 28:24
  are flowing through our platform to extract things like what the
• 28:27
  company is being invested in, like who's the investor, what's
• 28:32
  the amount that's being invested, what are all the
• 28:34
  terms, you know, is there a cap, like what's the valuation?
• 28:37
  And this is technology that we first applied internally to the
• 28:43
  flow of incoming documents, so we could help.
• 28:46
  Relations, you know, move faster, so they didn't have to
• 28:48
  review and read through like entire documents all the time.
• 28:52
  And we actually found that the models were super accurate,
• 28:57
  right? Look for data retrieval.
• 28:58
  Again, it's easy also to double check that whatever data is
• 29:01
  retrieved is actually in the document and we could match it
• 29:04
  against the data that's entered on our platform.
• 29:07
  When an investor actually submits a deal, they're able to
• 29:12
  add the key terms that they're investing in, and our technology
• 29:16
  was there. Would double check from the
• 29:18
  document that what's in the document actually matches what
• 29:21
  the investor expects. And so we naturally have the
• 29:25
  training set or a validation set to say that or extraction
• 29:29
  actually matches what is being inputted into our system and
• 29:34
  when it doesn't actually your extraction is correct and what
• 29:37
  was put in the system was wrong and so we could go back.
• 29:39
  It's actually a super valuable service and tell them, hey, you
• 29:42
  know there's a difference in the document from what you are
• 29:45
  expecting. You know your investment to be
• 29:47
  and. So from there, we actually
• 29:49
  decided to build a product dedicated to opening this
• 29:52
  capability to any investor, any fund manager or any investor
• 29:58
  LP's in the fund to analyse their own documents.
• 30:02
  And so that's a product called AngelList Relay that we shipped
• 30:06
  and opened to the public. There's a free tier, so you can
• 30:08
  go and try it like on five investment documents per month
• 30:13
  or five company updates per month.
• 30:14
  And what the tool does is essentially like with 0.
• 30:19
  Effort on your part. You just send or forward emails
• 30:23
  with the documents or the company updates that you're
• 30:25
  receiving and Relay will parse those, extract all the key
• 30:28
  terms, organise them by companies, right, and build a
• 30:31
  dashboard for you of all your investments, the terms at which
• 30:34
  you've invested. And then if you get company
• 30:36
  updates from those companies, will add them in the dashboard,
• 30:39
  you know, alongside the investment terms, so you can
• 30:41
  track how your companies are doing over time.
• 30:43
  And so, yeah, that's the product release from just like 2 days
• 30:47
  ago. Ohh so.
• 30:50
  I think if I'm understanding this correctly.
• 30:54
  I invest in a seed round and the company, I invest in 10
• 30:58
  different companies in their seed round right.
• 31:00
  And these companies all it's too much work for me to stay on top
• 31:05
  of everything that these companies are doing.
• 31:07
  But every now and again, hopefully they raise another
• 31:10
  round or they are giving me investor updates again,
• 31:13
  hopefully depending on the. Companies, I've, I've seen it
• 31:18
  all. But the idea here is that these
• 31:22
  companies when they give their investor updates, I can just
• 31:26
  forward that update to you or I or I can upload it to Relay and
• 31:32
  you will keep track of it And you will say, hey, will, you
• 31:34
  invested at this much AR and it was this valuation.
• 31:39
  Now they're saying that they have this much AR and they've
• 31:43
  got these many customers. And so you get to see the
• 31:46
  progress. That's exactly right.
• 31:48
  Yeah. And you can even give them your
• 31:50
  relay e-mail address so that they send the company updates
• 31:52
  directly to your dashboard. And we do this automatically.
• 31:55
  It doesn't matter what the format of the company update is
• 31:57
  like. Well, you know, parse it and
• 31:58
  structure it and extract the key information.
• 32:01
  And since we're relying on large language models like, we
• 32:04
  actually can figure things out, like if they talk about monthly
• 32:08
  burn, we can extrapolate it to like the yearly burn.
• 32:11
  Like if they talk about maybe negative values, it's easy to
• 32:16
  mark that as such. Click on the dashboard and if
• 32:20
  they're not sending updates like you can easily tell as well
• 32:23
  because again it's anchored on the investment documents that
• 32:25
  you've sent early on. And you can see this company has
• 32:28
  not sent us any updates in the last six months since our
• 32:31
  investment there. Maybe we should double check,
• 32:33
  you know, see what's up. So.
• 32:36
  What's going on over there? OK, so.
• 32:39
  Break it down. What exactly are you doing on
• 32:42
  the back end? How does the tech look?
• 32:45
  Yeah, so it's. Pretty much like what we've
• 32:47
  built internally for our own document classification and
• 32:51
  extraction use cases that we've packaged right on top of emails
• 32:55
  being sent away. So we have a really solid
• 32:58
  document infrastructure that allows us to store, securely
• 33:03
  store and analyse this information.
• 33:06
  And we've developed prompts that we run like Basically a
• 33:10
  cascading set of prompts like. We use length chain as the
• 33:13
  library to structure this and. The document comes in, we
• 33:18
  classify it, then from the classification we determine
• 33:20
  whether it's one of the documents where we support terms
• 33:23
  extraction. If we do, we apply like a pretty
• 33:27
  advanced prompt, like one that took us a lot of iterations to
• 33:29
  get right to parse all the key terms and key information and
• 33:33
  even maybe sometimes analyse the legal language for the
• 33:37
  documents. To flag anything that looks off
• 33:39
  for you know, probably would require an action on the
• 33:41
  investors part. And once that's all done, it
• 33:44
  becomes metadata that's attached to the customer's document.
• 33:47
  And that they can then retrieve in their portal sounds pretty
• 33:51
  straightforward because it actually is like from a
• 33:53
  technology perspective. And really the magic is in
• 33:58
  having fine-tuned those prompts with a lot of iterations to get
• 34:03
  the perfect output like what we expect customer like an investor
• 34:07
  to want to see when they send. You then you post when you were
• 34:11
  looking at the prompting to like figure out what the best prompts
• 34:14
  are. I know there's a bunch out there
• 34:17
  on that layer, like the prompt. Player to figure out how.
• 34:21
  How can we know which prompts are the best?
• 34:23
  Prompt Player, I think, is actually one of those companies,
• 34:25
  right? There's a company called
• 34:27
  promptly. That's.
• 34:28
  True, we've tried a few things. We also have like an internal
• 34:32
  system that actually supports all of our operations.
• 34:36
  That makes it really easy actually, to keep a human in the
• 34:39
  loop in this process where you upload the document, you apply
• 34:42
  the prompt to it, get the data, and then you can review the
• 34:45
  output then and there. And so yeah, I think for now
• 34:49
  it's still a bit artisanal. For, you know, it's like
• 34:52
  basically people doing some manual work we don't have.
• 34:58
  Fully structured regression testing capability in the sense
• 35:01
  that you could, you know, rerun a bunch of documents that we
• 35:04
  know should give this output and see if your new prompt leads to
• 35:08
  the same output. But this is in the works.
• 35:10
  Like these are definitely capabilities that we want for
• 35:12
  our system. It might come out that's like
• 35:15
  maybe an expensive test suite, but it's probably worth it that
• 35:18
  scale to make sure that we keep. Wretched.
• 35:22
  Seeing the quality up, not introducing any regression in
• 35:25
  the quality of our parsing O Tibo.
• 35:28
  Should I just become a prompt engineer?
• 35:30
  Is that what I should do for a living now?
• 35:32
  So it's a good question. The main call out from my part
• 35:36
  is actually that there's still a lot of plumbing, a lot of
• 35:40
  engineering work around the prompts, right?
• 35:42
  To get, for example, the documents you know formatted in
• 35:44
  the right way to standardise them, to extract the text from
• 35:47
  them. Like.
• 35:48
  That's a lot of steps until you can actually apply the prompt
• 35:51
  to. The document and even then some
• 35:53
  of the documents are super long, so you need chunking little.
• 35:57
  Working with length chain is not just prompt engineering, it's
• 36:00
  also like engineering data pipelines.
• 36:04
  And so I think there's still a lot of work to be done that's
• 36:07
  not prompt engineering. Another strategic move or or
• 36:12
  stands as a company is that a lot of the prompt engineering
• 36:14
  can be done by the people who know the business the best or
• 36:18
  the domain the best, right. So if the lawyers on the team
• 36:21
  can actually. Help tune and improve the
• 36:24
  prompts, right? That's ideal for us because this
• 36:27
  is where we get the highest leverage.
• 36:29
  And since prompts are natural language, like, it's actually,
• 36:33
  the barriers are gone. It's possible for anyone on the
• 36:36
  company to write a prompt and, you know, keep it, trading on it
• 36:40
  to get the output that they're expecting.
• 36:42
  And so we're also investing heavily in putting prompt
• 36:47
  engineering in the hands of operations and you know, the
• 36:52
  lawyers. The team, for them to be able to
• 36:55
  help us move faster, right, so that engineers can focus on
• 36:58
  what's the prompts are in a good space, deploying them in
• 37:02
  production and running them at scale.
• 37:04
  So I want to kind of hit on this a bit because interesting
• 37:07
  things, it's something I believed in for a lot of years.
• 37:09
  So like, I think obviously a human in the loop is important
• 37:13
  for this process, right? It's important for your guys's
• 37:16
  whole development life cycle and you're trying to work towards an
• 37:21
  automated regression test system to be able to, OK, can I, can I
• 37:25
  deploy without with less human intervention?
• 37:28
  But do you foresee no human intervention?
• 37:31
  And I preface this with like going back like 5 or so years
• 37:34
  when Auto ML was the big thing, right?
• 37:36
  Where everybody's all the rage about, hey, I could just throw
• 37:38
  some data things, little bit the ball for you, all the
• 37:41
  hyperparameter tuning for you and everything.
• 37:43
  So do you foresee that eventually we're not going to
• 37:46
  need a human in the loop in this process potentially?
• 37:50
  Like the process of drafting the prompts.
• 37:52
  And yes, like is prompt engineering going to be an
• 37:54
  automated thing? Pretty much.
• 37:56
  Yeah, I mean, if if you read Asimov, like, you know,
• 38:00
  eventually that's really where humanity gets at some points.
• 38:03
  And so, yeah, I think there's definitely.
• 38:07
  A long term vision where this is true and this, you know,
• 38:09
  happens. I think there is still quite a
• 38:12
  few. Definitely months, maybe years
• 38:15
  of really exciting work for for people, you know, for us to do
• 38:19
  before we get there. And we also need to still.
• 38:24
  Distil what actually is needed. You know what the customers
• 38:28
  want, what's terms like are relevant and important to parse
• 38:32
  from these documents. You know what's how do you see
• 38:35
  if there's a pay to play clause in an investment document,
• 38:38
  right. These are concepts that
• 38:41
  obviously I think at some point could be inferred automatically.
• 38:46
  But for now we're still doing the goal setting.
• 38:48
  You know, I haven't seen models that actually tell you.
• 38:51
  This is what you should do, you know, in a way that is directly
• 38:54
  usable and so still a few years of exciting work in this space
• 38:58
  for for us humans. So I wonder about when it comes
• 39:02
  back to the architecture of what you've built and.
• 39:07
  Having full confidence in the output, what are you doing to
• 39:11
  make sure that? There's not hallucinations that
• 39:15
  sneak through. Yeah.
• 39:16
  So for us in the problem set that we've tackled and actually
• 39:20
  deployed in production, it is possible to just double check
• 39:23
  that the extracted text is actually present in the
• 39:26
  document, right. So we always have ground truth
• 39:30
  that we can go back to. We were talking about how data
• 39:33
  is important in a given space. And I think it's still relevant
• 39:35
  and true for using large language models or foundational
• 39:39
  models because as I said like early on, we actually have a
• 39:43
  tonne of documents that have flowed through our platform for.
• 39:47
  Past investments and that are matched to the structured data
• 39:50
  in our database, right where we are, the system of record and
• 39:54
  that data has been double, triple checked by the user, by
• 39:58
  our team like the operations team and so that made for a very
• 40:02
  convenient. Like a test set, right.
• 40:04
  So we can write the prompt and check that what it extracts
• 40:07
  actually matches what's in our system and do that at scale,
• 40:11
  like with a lot of examples. And that's helped us build great
• 40:14
  prompts and also the confidence that the model is actually
• 40:17
  performing amazingly well. I think we're at the current
• 40:23
  stage, we're claiming, you know, 99% accuracy in Relay.
• 40:27
  This has BeenVerified like with all of the back testing that
• 40:30
  we've done on on previous documents so.
• 40:33
  I think both. These cases that you've talked
• 40:35
  about? You are in a great position
• 40:37
  because if things do go off the rails, you're not going to like
• 40:42
  kill anyone like in autonomous driving.
• 40:44
  I think in the Relay one it's probably more.
• 40:48
  Important that you do have that cross check with your database
• 40:51
  because it's money at play, right?
• 40:54
  And so maybe people in. Be getting an update that says
• 41:00
  they need to pay money and really they don't need to or
• 41:03
  whatever that may be. So it's good to have that there.
• 41:05
  But again, when I think about like the.
• 41:09
  Risk. On this use case, it is, it's a
• 41:13
  little bit lower down. When you think about something
• 41:16
  like Ohh well, is it the healthcare use case or is it an
• 41:20
  autonomous driving use case? And those are a lot more.
• 41:25
  Scary when you're just leaving it all up to a computer to
• 41:29
  figure out. And so I like that you've been
• 41:32
  able to figure out how can we incorporate this into our
• 41:36
  product as much as possible and. I imagine it's not all flowers
• 41:43
  and tie dye with the hippies. Peace and love.
• 41:45
  What have the challenges been, right?
• 41:47
  Like, there's gotta have been some snags that you ran into.
• 41:51
  Yeah, it's actually funny. It's really more around scaling
• 41:55
  or calls to Open AI. Like honestly, like the the
• 41:58
  biggest struggle for me has been spending time trying to figure
• 42:02
  out how to get access to GPT 4 to GPT 432K, how to get higher
• 42:07
  rate limits so that we can handle the volume right of
• 42:10
  documents that we want to parse. And it's funny because it's like
• 42:16
  a lot of it is trying to find the right point of contact at
• 42:20
  Open AI and getting a response other questions like that.
• 42:23
  And we, yeah, we found, you know, eventually found partners
• 42:28
  that could help us with this. We also found that we could
• 42:31
  leverage Azure directly. They posed exactly the same
• 42:36
  endpoints and it's actually a lot more flexible from you know
• 42:40
  scaling perspective and it's maybe a little bit closer to
• 42:42
  what we were familiar with, right, Like a cloud environment
• 42:46
  where you provision resources and you can increase rate limits
• 42:49
  with a simple request and you can track your usage in a way
• 42:53
  that's maybe a little more stable and structured.
• 42:56
  And so we now have calls going to both Azure and Open AI
• 43:02
  directly and like we're able to increase our rate commits by
• 43:06
  having that routing happening between the two.
• 43:08
  And we're also exploring deploying our own train models
• 43:13
  on top of these tasks. So I think costs are way lower
• 43:17
  than we had people doing this work.
• 43:19
  So we're already saving money by doing this and we expect the
• 43:22
  cost of intelligence to keep going down over time.
• 43:26
  And so we bet heavily that we wanted to 1st go with breadth,
• 43:30
  try to apply this to as many use cases as possible.
• 43:33
  Not thinking too hard about optimization just yet, but we
• 43:36
  obviously need to get there. The ability to maybe train their
• 43:40
  own model, deploy it on dedicated hardware so we can
• 43:43
  scale according to our needs, I think is an interesting next
• 43:46
  step for us, something that we're looking forward to.
• 43:49
  So is that like your upper limit then?
• 43:51
  Is that you're scaling upper limit?
• 43:52
  Is the third party vendor you're?
• 43:56
  That's your OK so that's what you're deaf.
• 43:58
  Yeah. I was gonna ask that it's from a
• 44:00
  challenge perspective. Do you think then there's that's
• 44:03
  why there needs to be investment in other models besides you know
• 44:06
  what opening eyes providing you do you do you find that there
• 44:09
  can be value and maybe we should diversify our model portfolio in
• 44:12
  terms of what we're using and use different models to even
• 44:16
  open that up and or I don't know kind of be flexible that way.
• 44:20
  Yeah, makes perfect sense and I believe so like pretty strongly.
• 44:24
  I think again ownership of the models themselves, you know
• 44:27
  would grant us more flexibility, right.
• 44:29
  We were talking about the 30% use case, this investment and
• 44:32
  maybe longer term capabilities where we're training bespoke
• 44:37
  models for each customer on their platform.
• 44:40
  From a technical perspective, it is doable, not super cheap, but
• 44:43
  it's not super expensive either. But it's only possible if we own
• 44:47
  the keys to the model layers, right.
• 44:50
  So if we own the base model, maybe have a layer that's
• 44:52
  specific to Angel list as a whole, like the investment, you
• 44:55
  know, ecosystem and then on top of that maybe a really small
• 44:58
  sliver that's constantly retrained for your own specific
• 45:02
  customer data. We in order to build something
• 45:05
  like this, we need control over how the models are trained,
• 45:07
  right. So there's some appetite to
• 45:10
  actually build up a competency in this space, right, and have
• 45:15
  on premise like trained internally trained models that
• 45:19
  we can work with. But as far as models that are?
• 45:24
  More or less the same idea of open AI.
• 45:27
  You haven't thought about going with like a coher or anthropic
• 45:31
  or anything like that. Like for us, having to integrate
• 45:34
  with another model would just, you know, increase the
• 45:36
  complexity of our system, right? And we figured that it was maybe
• 45:40
  not required to pay that cost just yet because we're not
• 45:43
  hitting any roadblocks with Open AI.
• 45:45
  It does what we need, right? So we're not in a space where
• 45:48
  we're going to save pennies by trying maybe a different
• 45:51
  provider just yet. And maybe another thing is
• 45:54
  there's a tiny bit of a relationship with Open AI,
• 45:57
  right? Like I guess Sam Altman spoke at
• 46:00
  the last Angeles Confidential conference.
• 46:02
  And so you know, and I'll say from personal experience, Open
• 46:08
  AI has been the top quality model.
• 46:11
  So we decided to just go use it for now, learn as much as we
• 46:15
  could in the space with the best in class capability, right.
• 46:19
  And then there's a moment in time when we're going to switch
• 46:21
  to cost optimization. And then I think it might be
• 46:24
  reasonable us to explore other approaches.
• 46:28
  Yeah, I I do like that idea. It's like, let's not get
• 46:32
  distracted with anything except for how can we bring AI into our
• 46:38
  product and how can we make sure that it's valuable And all of
• 46:42
  the other stuff. We can figure that out once
• 46:45
  we've proven out this first assumption, which is, does this
• 46:49
  actually make a difference? That's exactly right, Yeah.
• 46:53
  Well, I admire the focus on that because I am the opposite
• 46:57
  sometimes. So do you feel like there are
• 47:01
  ethical concerns that we should be thinking about when it comes
• 47:04
  to using AI and money? You talked about like the
• 47:09
  comparison with maybe the, you know, healthcare space or maybe
• 47:12
  self driving, right. I think the key here is that
• 47:16
  ultimately at the end of the day, like the legal documents
• 47:19
  are the source of truth, right? So there's ever a question with,
• 47:23
  you know, what the allocation in the deal should be or how much a
• 47:27
  certain person owns in that deal.
• 47:28
  You can always refer back to the the documents, right, the legal
• 47:32
  documents. And so I think from here, like
• 47:34
  in our space, as you said, like this is an industry that's like
• 47:38
  a perfect fit for these models because we're talking about
• 47:41
  text. It's all text, right?
• 47:43
  A lot of legal text. So it's also pretty structured,
• 47:46
  right? It's not completely free form.
• 47:49
  And yeah, the outcome, like the worst case scenario is, you
• 47:54
  know, just something where something looks off and you go
• 47:56
  double check the document and like you use that as the guiding
• 48:00
  truth. And so it saves A tonne of time
• 48:02
  from people, humans reading through a lot of text that's
• 48:07
  really repetitive. And as I said, humans also make
• 48:11
  mistakes. And we found that actually the
• 48:13
  models are more accurate and make fewer mistakes than when
• 48:17
  you have people even like experts reading through those
• 48:20
  documents, right? Sometimes you just go maybe too
• 48:22
  fast or you skip paragraph that had the information that you
• 48:25
  needed. The model won't do that right.
• 48:28
  It will look at everything in the document and consistently
• 48:31
  output the same results from that analysis.
• 48:35
  Yes. So here's another one that I
• 48:37
  totally forgot that I wanted to ask you and but when it comes to
• 48:40
  prompts, have you noticed that? When Open AI releases a model
• 48:45
  update, it breaks some of your prompts or you have to go and
• 48:49
  rework them. This happened to us, so it was
• 48:53
  temporary though like I I need to clarify that this was like
• 48:56
  the move to the June model basically.
• 48:58
  Yet there was maybe an 8 hour period where things were broken.
• 49:03
  So technically like our API calls were either failing or
• 49:06
  maybe using the wrong model and that got fixed and now we're
• 49:11
  back to where things were before.
• 49:12
  We also found that maybe some prompts are actually working
• 49:15
  better than they used to. I know there's a lot of noise
• 49:18
  out there about, you know how there's a regression in the
• 49:20
  quality of the output from from GT4, especially for things
• 49:24
  around code. We're not parsing code like
• 49:26
  we're doing legal document reading and entity extraction,
• 49:30
  and for those, use cases like this has been a continuous March
• 49:33
  towards better accuracy with simpler prompts.
• 49:38
  Yeah. The part about maybe operational
• 49:41
  excellence like still being a work in progress is something
• 49:45
  that like does ring true to me, like the models like or open
• 49:51
  API's can't go down, right. And so we've built some
• 49:55
  resilient systems that can retry until they're back up.
• 49:58
  It's less true now than it was three months ago.
• 50:01
  But yeah, I'll just say that having worked now with Azure
• 50:04
  Open AI and Open AI APIs directly like Azure is actually
• 50:07
  more robust and has been more consistent in its responses.
• 50:12
  But the underlying models are the same.
• 50:14
  And so right now we're load balancing between the two, but
• 50:17
  we actually are glad that we have a fall back if the main
• 50:19
  models, you know, start breaking or like not responding anymore.
• 50:23
  Yeah, yeah. Maybe another incentive to grow
• 50:26
  and build in house, right. So that like we're not dependent
• 50:29
  on third parties availability. That's said.
• 50:32
  Again, plus priorities. Yeah, that comes with its own
• 50:37
  challenges. Would you start building your
• 50:38
  own stuff? Right.
• 50:39
  Then we might also hit operational excellence issues,
• 50:43
  so. Exactly.
• 50:45
  Exactly. We'll do.
• 50:46
  This has been awesome. I really appreciate you coming
• 50:48
  on here and giving us a little peek.
• 50:51
  Behind what's happening at Angel List and how you all are using
• 50:55
  this and how you've been up, been able to get up and running
• 50:58
  so quickly, it's fascinating to hear the story and I think we'll
• 51:03
  end it there. Sounds good.
• 51:06
  Yeah. Hey, I'm Vishnu.
• 51:13
  I'm a data scientist at first hand and I definitely think that
• 51:16
  you should subscribe to the ML OPS Coffee Sessions podcast.
• 51:20
  It's the best podcast out there to stay on top of what ML OPS
• 51:23
  actually is to talk to the true thought leaders in the space.
• 51:27
  And oh by the way, Demetrius is absolutely hilarious.
• 51:30
  What a weird guy. You should definitely subscribe
• 51:32
  to the podcast.