Latest / Agentic Conversations (formally mlops.community) / Relational Foundation Models: Unlocking the Next Frontier of Enterprise AI // Jure Leskovec // #348

Transcript

Relational Foundation Models: Unlocking the Next Frontier of Enterprise AI // Jure Leskovec // #348 · Agentic Conversations (formally mlops.community)

• 0:00
  They, people don't even realize or we seem to have forgotten how
• 0:04
  valuable structured data is because it's kind of the ground
• 0:08
  truth of the business and it's it, it, it captures the
• 0:12
  blueprint the of the business. And we seem to have forgotten
• 0:16
  all that. And you know, we are now excited
• 0:18
  about answering questions about documents.
• 0:26
  Now this is a bold claim. Being able to create machine
• 0:32
  learning models faster and better.
• 0:35
  What is this? Break that down for me a little
• 0:38
  bit more. Yes, I can.
• 0:41
  I can, I can give you an, I can give you an explanation, right?
• 0:45
  If you think about how we are building that it is predictive
• 0:49
  machine learning models today, right?
• 0:50
  This, this models would be let's say churn models, recommended
• 0:54
  systems, any kind of risk scoring models, fraud models
• 1:00
  across all kinds of industries, you know from from hospitals to
• 1:03
  social media to manufacturing to supply chain optimization and so
• 1:07
  on. The way we are doing this is
• 1:09
  right that fundamentally these models use the internal data
• 1:16
  that is usually structured in tabular form, right?
• 1:19
  It sits in some database in some data warehouse.
• 1:22
  It usually has this relational structure.
• 1:24
  You have multiple tables interlinked with primary foreign
• 1:28
  key relations, right? So it would be a user table, a
• 1:31
  product catalog and and record of all the orders of these
• 1:36
  users, maybe the the table that includes all the website
• 1:38
  behaviour. So all the clicks that says this
• 1:40
  user clicked on this web page at this time and so on, right?
• 1:44
  And all the features we want exactly, exactly right.
• 1:47
  So this would be the data and the way we build these models,
• 1:50
  these models today is that we would basically now be joining
• 1:53
  these tables, generate features and then train, you know, our
• 1:56
  favorite neural network or whatever it is on this data.
• 2:01
  And I can go deeper. What what I mean by by by all
• 2:04
  this, but basically takes us, I know, several months to build
• 2:07
  these models. We need to engineer the
• 2:09
  features. Features need to be up to date.
• 2:12
  They must, must not be stale. There is this time travel issue.
• 2:15
  We then put these models, let's say, in production to actually
• 2:18
  run more than just in our private notebook.
• 2:20
  And that's another can of worms and so on.
• 2:23
  But if you look at what's kind of been the trend in AI, let's
• 2:26
  call it this way, right? The trend in AI has been let's
• 2:29
  learn on the raw data. OK, so in big revolution in
• 2:33
  computer vision was let's not feature engineer from the pixels
• 2:36
  in the image. Let's train a neural network
• 2:39
  directly on the pixels, OK, And we get this super great neural
• 2:43
  networks for computer vision, total revolution.
• 2:46
  Now we are in the age of large language models, right?
• 2:49
  In the old days when we did let's say natural language
• 2:52
  processing, natural language understanding.
• 2:54
  Even if you think when you know, IBM won Jeopardy, the
• 2:57
  competition, it took 300 people two years and a bunch of very
• 3:03
  careful feature engineering. So that computer was able to
• 3:06
  answer Jeopardy questions and be better than humans.
• 3:10
  So in some sense we had AI at that time, but it was just kind
• 3:13
  of hand engineered, right. Today, Transformers kind of they
• 3:16
  just learn over a sequence of tokens and it's kind of they
• 3:21
  learn over raw data. So what the interesting thing is
• 3:24
  to ask, OK, so it seems like that the world of machine
• 3:27
  learning, let's say predictive modeling has been stuck in the
• 3:31
  past. We don't learn on the raw data,
• 3:33
  we learn on this featurized data.
• 3:36
  So what I'm, what I was very excited about was, was to say,
• 3:40
  OK, how could we change this? How can we develop methods that
• 3:43
  can learn directly on the raw data?
• 3:46
  And the hard part here is that your, your data in machine
• 3:50
  learning does not come from a single table.
• 3:52
  It comes from multiple tables, right?
• 3:54
  So. And it's usually very messy too,
• 3:56
  right? It's usually not pristine data
• 3:59
  that gives you a clear story. It's not, it's not pristine.
• 4:02
  It has missing information and all that, right.
• 4:05
  So what we have developed is a set of neural networks and I can
• 4:09
  talk about them. They are based on, they are
• 4:11
  basically based on the graph representation of the data that
• 4:14
  allow you to learn directly on the raw database data.
• 4:17
  So they allow you for the first time to learn over multiple
• 4:21
  tables at once and now because of that you completely sidestep
• 4:27
  the feature engineering process, which means you save humongous
• 4:31
  amount of time. And.
• 4:33
  Compute and compute. Compute maybe not so much
• 4:37
  actually you need GPU's because these models are now a bit
• 4:40
  bigger. So, so you know, you need, you
• 4:42
  need to train on GPU's, but you know, the computer works when,
• 4:45
  when you sleep. So, so it's, it's very fine.
• 4:48
  But there is another benefit, which is because now the neurons
• 4:52
  are learning how to combine the data, the fidelity, the the
• 4:57
  nuance of those loading signals, it's much, much higher.
• 5:00
  So your models get more accurate.
• 5:05
  So now I've heard folks talk about recommender systems with
• 5:09
  LLMS and just basically the inherent understanding that LLMS
• 5:14
  have. You can suggest certain things
• 5:19
  to an LLM and it's going to be able to have a better view of
• 5:23
  what a user will want. Is this kind of what you're
• 5:27
  talking about, or is this a whole separate thing I.
• 5:31
  I would say it's a separate thing, right?
• 5:33
  Like if you want through LLMS with for recommendations, it
• 5:40
  kind of works OK. It's not super terrible, but
• 5:46
  it's not good either because it's all kind of based on some
• 5:49
  kind of common sense. It's not really learned from the
• 5:52
  data for your specific users, for your specific understanding
• 5:55
  and so on. And I can give you a simple
• 5:58
  story. For example, I was chief
• 6:00
  scientist with Pinterest for six years, right?
• 6:02
  Took Pinterest from 100 people to post IPO and I built a
• 6:07
  recommender system platform there 3 generations.
• 6:11
  And what was for example a super hard problem with with Pinterest
• 6:14
  is that as you see these images, maybe you have an image of the
• 6:18
  rug on the floor and you have an image of a tapestry on the wall.
• 6:23
  They practically look the same. The neural network could not
• 6:26
  tell the 2 apart, but humans could.
• 6:30
  And the behavior on those things were was very, very different,
• 6:34
  right. So at Pinterest, we very quickly
• 6:35
  realized it's not just enough to treat a user as a sequence of,
• 6:39
  let's say a sequence of images they visit because the neural
• 6:43
  network would get confused. What you need to think of it is
• 6:46
  a is a. Basically, you need to think of
• 6:48
  Pinterest as a set of interactions of users with
• 6:51
  content and think of that as a graph.
• 6:53
  So if you say, oh, if I, if I interact with this image, and
• 6:56
  since other people interacted with it as well and you know,
• 7:00
  then I kind of know what this image is about, right?
• 7:02
  This, this tapestry can say, oh, I'm interacted with other
• 7:06
  tapestries. So I'm a tapestry.
• 7:07
  I'm not a, I'm not a rug on the floor.
• 7:09
  And a rug on the floor says, oh, when people click on me, they
• 7:12
  also explore other things that very much look like rugs.
• 7:16
  So I know myself, right? I'm not a, I'm not a tapestry.
• 7:20
  I am a rug, right. Like it's in this kind of sorry.
• 7:24
  It's good by association. Yeah, exactly, exactly.
• 7:28
  And this is what this kind of graph based approaches allow you
• 7:30
  to do because they allow you to kind of learn from this
• 7:33
  structured relational relational information.
• 7:36
  And this is what sequence sequence models sequence models
• 7:39
  will miss. And if I'm understanding this
• 7:43
  correctly, for example, when you're doing the traditional
• 7:47
  machine learning models, you're really optimizing for one or two
• 7:52
  things. And with what you're proposing
• 7:56
  now and what you've built it, does it have a more holistic
• 8:00
  view? Like if we have the two extremes
• 8:02
  where you have traditional ML models and then the other
• 8:06
  extreme is a large language model, where do you fall on that
• 8:11
  spectrum? That is a good question.
• 8:15
  So the approach we developed, it's called the relational deep
• 8:20
  Learning and it can take any database represented as a graph
• 8:25
  of relations between the entities in your database.
• 8:27
  And now we just learn over that graph of relationships.
• 8:32
  And if you now train a transformer like architecture
• 8:35
  that is called a relational graph transformer that does not
• 8:39
  attend kind of over the tokens like a typical transformer, but
• 8:43
  attends over the tables of that of that schema of the database.
• 8:48
  You can you can do both. You can train small task
• 8:52
  specific models, kind of you can think of them as you know, kind
• 8:56
  of classical machine learning models, but now really learned
• 8:59
  directly from the raw database data.
• 9:01
  But you can also learn large pre trained model that is good at
• 9:05
  any task on any database. So you write like if you go to
• 9:10
  che GPT and ask it, Hey, do time series forecasting for me or you
• 9:14
  go and say, Hey, I have this transaction, how likely it is to
• 9:17
  be fraud you, you will get terrible results, right?
• 9:20
  How would Che GPT know the likelihood of a particular
• 9:24
  transaction being fraught? It won't, right?
• 9:27
  But the the large pre trained relational graph transformer,
• 9:31
  which we call a relational foundation model allows you now
• 9:34
  to ask this type of questions without any model training.
• 9:37
  So maybe the point is the same way as in let's say in biology
• 9:43
  we accept that we need adna foundation model, we accept that
• 9:46
  we need a protein foundation model because protein is not
• 9:49
  natural language. I think the key thing we need to
• 9:53
  accept is that for structured data, we need foundation models
• 9:57
  trained and built for structured data.
• 10:00
  We cannot textify a database, throw that into an LLM and and
• 10:05
  hope it will work because it does not.
• 10:09
  OK, so I'm starting to understand a bit more of the
• 10:12
  picture of what you've built and now can we get into the nitty
• 10:15
  gritty of how this actually is built and what you're doing?
• 10:20
  Do I just throw in my database and then do I need a a whole ton
• 10:26
  of data? It's like what?
• 10:28
  What does this look like in practice from how I how I
• 10:32
  interact with it, or how I train a model that is now going to be
• 10:36
  one of these types of models? Yeah.
• 10:39
  So the way the way you need to do is you, you need to let's say
• 10:43
  first select the tables. The scheme I want to learn over
• 10:49
  usually you know, it's 5 to 50 tables and what you need to do,
• 10:55
  you need to specify what are the primary foreign key relations
• 10:58
  between these tables, right. That user ID in this table is
• 11:01
  user ID in that other table. And you select kind of the
• 11:05
  semantic types of columns. So there is a bit of let's say
• 11:07
  data registration, data modeling, even you can call it
• 11:11
  data preparation step, but it's it's very small because you just
• 11:15
  need to say this is my schema here are the relationships.
• 11:20
  And after that is done you have two options.
• 11:24
  One option is that you use now the large pre trained model that
• 11:29
  you just point to this collection of tables.
• 11:31
  You prompt it with a specific predictive question and half a
• 11:35
  second later you will get an answer that is on average as
• 11:41
  accurate as a manually built model that takes, I know a month
• 11:44
  to build. What you can also do is then to
• 11:49
  say, OK, I am going to fine tune my model for a specific task and
• 11:53
  for a specific database. And then you can fine tune and
• 11:57
  that will get you to superhuman, superhuman performance, right?
• 12:01
  Because now you have specialized the model for specific database
• 12:05
  for a specific task. And of course if you are for
• 12:09
  example doing fraud detection, which is a high value task where
• 12:13
  you truly care about, you know, the last bit of performance
• 12:16
  because it's saving you so much, let's say you're in money, then
• 12:20
  you want to train a task specific model.
• 12:24
  If you are in a more of a regime where you want to do basically
• 12:27
  ones like ad hoc querying, ad hoc predictive querying, you
• 12:31
  would use a large pre trained model to do that.
• 12:35
  And on these task specific models, do you see folks
• 12:39
  creating an ensemble of models just like you wouldn't maybe in
• 12:43
  the traditional fraud detection use cases?
• 12:49
  Not, not really, not really what what what we basically see is
• 12:54
  people training a single, a single model that then learns
• 12:59
  how to attend over all these, let's say different data from
• 13:03
  transactions to locations to time and all that and combine
• 13:07
  that into the best possible predictive signal, right?
• 13:10
  That's the difference is if you think about maybe a fraud model
• 13:14
  in a fraud model, you say, OK, I have AI have a customer.
• 13:17
  How many transactions did they do last week?
• 13:20
  And you create a feature and then somebody, you know, some
• 13:23
  other data scientist wake wakes up and says, it's not number of
• 13:26
  transactions last week is the number of transactions in the
• 13:28
  last, you know, 10 days. Great, let's add one more
• 13:31
  feature. Then somebody else wakes up and
• 13:33
  says, no, no, I, I, I know the answer.
• 13:36
  It's how much you spend in the morning.
• 13:38
  And I define morning as I know 6:00 to 9:00 AM and let's add
• 13:42
  that counter there. And then somebody else wakes up
• 13:45
  and says, hey, but there is daylight savings time.
• 13:47
  We just change the clock in US. OK, So you know, in the in the
• 13:50
  summer, this is the morning, in the winter, that's the morning.
• 13:53
  But you see my point kind of the point is that it gets kind of,
• 13:56
  we are computing this arbitrary statistics over the data to, to
• 14:01
  be able to say, OK, how much did this person purchase in the last
• 14:05
  time period? But if you have the attention
• 14:08
  mechanism, the attention mechanism can attend over each
• 14:11
  individual transaction. So it can learn to combine them
• 14:15
  in ways that humans will never, never will.
• 14:20
  And that's why it's able to kind of extract more signal from the
• 14:24
  raw data than, you know, a sequel query or a manually
• 14:27
  defined feature will. Because the neurons can do can
• 14:32
  do so much more. And they are basically trained
• 14:35
  to combine the data together, if that makes sense.
• 14:39
  Yeah. It's basically like move 37 in
• 14:42
  the go and how you know in go how what was it?
• 14:48
  It was move 37, right. I think you would.
• 14:50
  Exactly, exactly, exactly. It's maybe something like this.
• 14:52
  It can be counter intuitive. It can be something that is very
• 14:55
  fine grained. It could be some, you know, 2nd
• 14:58
  order correlation that you as a data scientist, as a machine
• 15:01
  learning engineer never put into a feature and things like that.
• 15:05
  And because of that, you know, it's, I would say the same
• 15:08
  intuition as in computer vision, right?
• 15:10
  In computer vision, we are like, oh, here's the edge, here's the
• 15:13
  gradient of color. Nobody's doing that anymore.
• 15:16
  It's just the neural network figures it out how to combine
• 15:19
  all the pixels to say what's in the image.
• 15:21
  And, and basically we do the same, right?
• 15:24
  We take the database, represent it as a graph, and then the the
• 15:29
  transformer architecture. The graph transformer
• 15:31
  architecture learns how to combine all these pieces of
• 15:34
  information into an accurate prediction.
• 15:37
  Rather than you know what is done today with machine
• 15:40
  learning, which is you manually figuring out how to combine
• 15:42
  transactions into something that allows you to predict something
• 15:46
  about the user. Yeah.
• 15:48
  So you're wiping away all of that feature engineering.
• 15:53
  What are U.S. Data scientist and machine
• 15:57
  learning engineer is going to do now?
• 16:00
  That's a great question. I think it's exciting.
• 16:03
  I think it's good because you're not a feature engineering
• 16:06
  machine, right? It's almost like saying, oh, you
• 16:10
  brought me a robot that helps me sweep.
• 16:12
  But I, you know, I want to be sweeping the floor all the time.
• 16:15
  No, right. Like you, you are here to to
• 16:17
  build models to to have impact on the business.
• 16:21
  You're not here to clean data and feature engineer, right?
• 16:24
  Those are the two most boring arts of your job.
• 16:27
  O what we see for example, is that this really kind of opens
• 16:31
  the data scientists to be able to build models faster, to
• 16:35
  refocus on modeling, to focus on what should we be even
• 16:38
  predicting, right? You never in, in let's say in
• 16:41
  business, in enterprise, you never come there is never a
• 16:44
  predictive problem that is given to you.
• 16:46
  It's a business problem that is given to you.
• 16:48
  So how do I go from a business problem to a predictive problem?
• 16:51
  Asking yourself what is even predictable?
• 16:54
  What is modelable from my data, right?
• 16:56
  Because you know, garbage in, garbage out.
• 16:59
  So what we see is then, because now the, let's say the future
• 17:04
  engineering step, you don't need to worry about, you are much
• 17:08
  more worrying about what's the raw data, How is the raw data
• 17:11
  structure? What is the information flow
• 17:14
  over the raw data and so on, right?
• 17:16
  And, and maybe let me just say this because I get asked many
• 17:20
  times. People say, oh, is this auto ML,
• 17:24
  right? Like remember this old auto ML
• 17:27
  promise, right? It's what was auto ML, right?
• 17:31
  Auto ML is you run a gazillion of SQL queries against your
• 17:37
  database. You join every table with every
• 17:40
  table, you aggregate everything 7 different ways, you create
• 17:44
  this humongous silly feature vectors, and then you train a
• 17:49
  bunch of models and hope it will work right.
• 17:52
  So it's like kind of throwing spaghetti against the wall and
• 17:55
  hoping that something sticks. What what I'm talking about is
• 17:58
  fundamentally different. It's a single neural network
• 18:02
  that has the ability to learn to attend over the collection of
• 18:07
  tables. And you are you are now training
• 18:10
  or tuning this neural network on that collection of tables.
• 18:13
  So it's faster. It it scales to, you know, to
• 18:18
  the, to the largest use cases in the world.
• 18:21
  I can say more about that later and and it gives good.
• 18:26
  Performance. Well, one thing that is very
• 18:32
  clear after hearing this is the value of that raw data.
• 18:36
  As if we didn't know that already, right?
• 18:38
  But how much can you have messy, ugly, shitty data and still give
• 18:49
  value from this method basically?
• 18:53
  That's a great question. So here is what I what I would
• 18:56
  say. In some sense, quality of the
• 18:59
  data of the data matters. But because these models are
• 19:03
  learning from the entire relational structure, they can
• 19:08
  implicitly impute missing labels, They can correct, they
• 19:12
  can correct for mistyped or misentered information.
• 19:16
  Even if the linking information is a bit noisy, sometimes the
• 19:20
  models can recover from it because you know, they're kind
• 19:23
  of learning holistically across the entire relational structure.
• 19:27
  So another place, for example, that this is important if you
• 19:31
  think about cold start problems, right?
• 19:33
  Cold start in a sense that let's say maybe in the recommended
• 19:36
  systems and so on, or you have a new user, you have a new item,
• 19:39
  you haven't seen anything about it.
• 19:40
  But then you have an old user or an old item that a lot of others
• 19:44
  have already interacted with. So you are very data reach,
• 19:48
  information reach about this item.
• 19:50
  What is the benefit of these models is that they can kind of
• 19:53
  trade off if the item is new, they will focus on the
• 19:57
  attributes of the item, properties of the item.
• 19:59
  I know description, image and things like that and they latch
• 20:02
  on that. But as soon as that item starts
• 20:05
  getting connected into the graph by users interacting with buying
• 20:09
  it and so on. Now the model is starting to use
• 20:11
  this relational structure. You know what you kind of said
• 20:14
  earlier like the the Guild by association to say, OK, who is
• 20:17
  interacting with this? What else are these people
• 20:19
  doing? Let me give an accurate
• 20:21
  recommendation so. It allows a kind of work.
• 20:24
  The models learn how to trade off between learning from
• 20:28
  attributes to learn and learning from structure.
• 20:31
  Just it it leads to very robust models.
• 20:34
  That's all I wanted to say, both in terms of missing data, data
• 20:39
  ugliness, as well as robustness. Do we need to have super clear
• 20:46
  ontologies and a knowledge graph set up or is that something that
• 20:50
  it infers also? Good question.
• 20:54
  I would say somewhere in between.
• 20:57
  All we need to know is what table links to what table.
• 21:02
  Many times we can infer this automatically from column names,
• 21:07
  and that's about all we need. So we don't need.
• 21:10
  We don't. The model does not rely on
• 21:13
  semantic information, right? The model.
• 21:17
  The model relies on patterns in the data.
• 21:22
  So maybe here is the the the the.
• 21:24
  The difference is that the model learns how to recognize the
• 21:28
  recognize the patterns in the data to make those predictions.
• 21:32
  So we don't need a super rich semantic model.
• 21:35
  We don't need to explain what is the true meaning of every column
• 21:39
  to the last detail, because from the kind of the patterns from
• 21:43
  the past that predict the future, that doesn't matter so
• 21:46
  much. So talk to me about scale.
• 21:50
  Scale good question. So this this scales to tens 100
• 21:57
  billion billion notes. So for example, I can I can tell
• 22:00
  you maybe to just to illustrate, I can tell you some examples of
• 22:05
  how this technology runs in practice and who's using it.
• 22:08
  OK, so one interesting story was, you know, a few years back,
• 22:15
  DoorDash came to us, right? The DoorDash during COVID was
• 22:18
  growing great, right? Everyone is ordering food, but
• 22:21
  post COVID kind of right? So, so, so, so they they they,
• 22:25
  they came to us and said, hey, can you help?
• 22:28
  Can this technology help us? So we looked at the restaurant
• 22:32
  recommendations, right? So recommend restaurant you are
• 22:36
  going to order from next. In particular, recommended
• 22:39
  problem is order recommend a restaurant you never ordered
• 22:43
  from before. So it's try something new and
• 22:46
  this the way this gets surface to you is through notifications,
• 22:50
  right? You get a notification saying,
• 22:52
  hey, would you do you think you want to order from this
• 22:55
  restaurant today, right. And and this is one of the core
• 22:59
  problems at DoorDash, right. So, so they've been kind of
• 23:02
  using traditional technology to build these types of systems
• 23:05
  with the best possible people and so on.
• 23:08
  We trained our our transformer model over a collection of
• 23:13
  tables. It was a 30% increase in
• 23:16
  accuracy 30. Wow.
• 23:20
  So me as a user, when you suggest a restaurant for me to
• 23:26
  try, it's much more pointed and it's much more to my liking.
• 23:32
  Exactly. It was much more pointed, it was
• 23:34
  much more to my liking and that resulted in several hundreds of
• 23:39
  millions of more purchases of or orders on, on on DoorDash,
• 23:44
  right. So it was humongous business
• 23:46
  impact, right. So that's just one example,
• 23:50
  right. Then now you say what's the
• 23:51
  scale of DoorDash is? I know several 100 million
• 23:54
  users, I know hundreds, you know, billion orders, all the
• 24:00
  website behavior, all the searches, all the geographical
• 24:03
  location, all the cuisine information.
• 24:04
  So it gets quite, quite, quite interesting as well.
• 24:10
  That's for example, one example where we saw like this humongous
• 24:13
  lifting performance over a flagship model, right, something
• 24:17
  that is not one data scientist woke up and and said, oh, can
• 24:20
  these can these do better? No, it's like a team several
• 24:23
  years of effort, it can do better.
• 24:26
  I can give you another example that's even larger scale and
• 24:31
  this got this is in advertising. Oh, before, wait, sorry, Yeah,
• 24:36
  on that DoorDash one. How long did it take to go from
• 24:42
  inception to production with that?
• 24:44
  One benefit of these models is because they run on the raw
• 24:47
  data, putting them in production just means refresh the raw data,
• 24:53
  right? So, so for people who have put
• 24:55
  models in production, like just having that feature store,
• 24:58
  having up to So, so it's very hard, right?
• 25:15
  But here it's much easier because you just say, here's my
• 25:17
  French data, make predictions on it, you refresh the data, make
• 25:22
  predictions on it. So we put that in production, I
• 25:25
  would say quite quickly, you know, maybe a few weeks.
• 25:28
  And of course, and that was a set of.
• 25:31
  It's irrespective if the data schema changes, as long as you
• 25:36
  refresh the data. Even if the data schema changes,
• 25:40
  you're still able to infer the right things.
• 25:46
  If the data, so the models themselves allow for data schema
• 25:50
  changes, so it's, it wouldn't be a problem.
• 25:53
  But if you, let's say, drastically change the schema,
• 25:56
  change the composition of tables, you would, let's say,
• 25:58
  retrain the model. That takes maybe a couple of
• 26:00
  hours. And now we have and now you're
• 26:03
  back in the game. Yeah, right.
• 26:05
  So, so that's, that's one benefit.
• 26:08
  Another benefit is if you think about user behavior changes all
• 26:12
  the time, right? Or if you think about fraud,
• 26:14
  right, Fraudsters are always inventing new ways to, to, to
• 26:18
  commit fraud. And, and you as a, or we, as
• 26:21
  data scientists, we are always a step behind because we are like,
• 26:24
  oh, my model performance is deteriorating.
• 26:26
  I need new features, but with this approach you're just let me
• 26:30
  retrain the model and the model will figure out the new the new
• 26:32
  signal and and you know, and and and so on.
• 26:35
  So it's also much easier to keep up to date.
• 26:37
  You can automatically retrain and really get the most value
• 26:40
  out of the data you have. OK, it makes a lot of sense.
• 26:45
  So yeah, what's the other scale? The massive scale, one that's
• 26:49
  even bigger than DoorDash. What's bigger than Doris?
• 26:52
  I can talk about this one. It's advertising models, right?
• 26:56
  So this is the bread and butter of the Internet industry, right?
• 26:59
  It's like predicting how likely is a user to click on an ad
• 27:03
  because you know every time you visit the website, there is a
• 27:06
  prediction for this user what is the most likely ad they are
• 27:10
  going to click on that Ads get put in front of you.
• 27:12
  If you click it, the advertiser gets paid.
• 27:15
  So no click, no money click, you'll get money, right?
• 27:19
  So 1% lifting accuracy of this predictive model means 1%
• 27:24
  lifting your revenue. So the use case here is Reddit
• 27:31
  and Reddit has advertising models running on their website
• 27:35
  and you know, it's bread and butter of what they do and
• 27:37
  every, you know, single or 10th of a percentage point matters a
• 27:42
  lot in terms of absolute revenue.
• 27:43
  So there I can say they they usually increase the accuracy of
• 27:49
  the model by 1 to 2% year over year.
• 27:53
  And with this approach was like 5 years worth of improvement,
• 27:58
  inaccuracy within a month or something, right?
• 28:02
  Again, over this flagship model that's been tuned and you know,
• 28:05
  the latest from research incorporated and so on.
• 28:09
  And I think this just shows kind of the power of this.
• 28:12
  Just let the neural network learn over the data, right?
• 28:16
  The trick is the data is split or seats in multiple tables.
• 28:21
  So you need now a generalized transformer architecture that
• 28:25
  can attend across the tables and learn how to extract the signal
• 28:29
  automatically, rather than us humans doing it manually.
• 28:34
  Yeah, I was talking to my friend Sid Anthropic and he was saying
• 28:37
  that they have a saying internally which is let the
• 28:40
  model. Cook basically, basically
• 28:43
  exactly, exactly, exactly and and I think my point is for
• 28:49
  natural language human like tasks, LLMS are great, but LLMS
• 28:54
  totally fail on structured data, right?
• 28:57
  You cannot take a database and Jasonize it, put it in a big
• 29:01
  BLOB of Jason, put that as a prompt and say, you know, now
• 29:05
  based on these, give me a prediction.
• 29:08
  What will the user like? Do you see this working for time
• 29:13
  series also? I see this working for time
• 29:16
  Series A lot as well. Exactly.
• 29:18
  There are two ways to think of time series, right?
• 29:20
  One is just to say, oh, I have an individual time series, I
• 29:23
  have a sequence I and I predict the next token.
• 29:25
  So people have been I would say quite successful training
• 29:29
  sequence based Transformers on time series data.
• 29:34
  Our approach is a bit different because and the way we think of
• 29:38
  it is it's not one time series, it's a it's a graph of time
• 29:41
  series because this time series are usually connected, you know,
• 29:44
  so maybe you have what sales record for one for one product,
• 29:48
  but products are related to each other or maybe they are stores I
• 29:52
  know sold in different stores and things like that.
• 29:55
  So by representing this time series as a graph, you actually
• 29:58
  get a further increase in performance because the model
• 30:01
  does not only learn how to how to make prediction from the
• 30:04
  single time series, but also learns how to attend across
• 30:06
  other related time series to better forecast, right?
• 30:10
  Because some time series might have different legs, some
• 30:14
  products might be correlated, some stocks might be correlated
• 30:17
  and things like that, right? So predicting from a single time
• 30:20
  series is very hard because you don't have that information.
• 30:22
  But through this let's say graph based approach you can borrow
• 30:26
  learn to borrow information from other related time series which
• 30:30
  again leads to the increased performance increased.
• 30:33
  Feels like you're just giving a much richer picture for the
• 30:38
  model to understand. That is a great way.
• 30:42
  I think that is a great way to say it, right?
• 30:44
  You give a much richer set of information to the model, and
• 30:48
  then you give the neural network the freedom to combine this
• 30:52
  information in in the best possible way for that forecast,
• 30:56
  that prediction, that risk score.
• 31:00
  Another thing that I'm thinking about is does it take a team to
• 31:09
  make this happen or is this something that is like how many
• 31:15
  people, how many resources? If I wanted to try and do this
• 31:18
  at my company tomorrow, what am I looking at actually like
• 31:22
  needing to dedicate towards this?
• 31:25
  That's a great question. What we see is that with this,
• 31:29
  you need about 20X fewer resources than traditional
• 31:33
  approach. So it means you can do much more
• 31:37
  with a much more leaner team. You still, let's say need a data
• 31:43
  scientist or someone who who kind of understands predictive
• 31:45
  modelling, but that person can build, you know, 10 models in a
• 31:50
  single day if they like to do to do so.
• 31:53
  And then what you also need, I think with predictions, it's
• 31:55
  always important that you are able to plug them in into
• 31:59
  whatever product, whatever decision making you are doing,
• 32:03
  right. So predictions are only useful
• 32:05
  when you are making decisions, when you are making taking some
• 32:09
  action based on that decision. So we need some engineering
• 32:12
  resources that says OK, now that we have prediction, here is what
• 32:15
  we are going to do with them. Yeah, it's still the model
• 32:20
  building and then putting the model into production.
• 32:22
  It's just that the model building piece has been
• 32:26
  drastically cut. Exactly or the exactly the, the,
• 32:31
  the model, the time to build a single model has been cut, which
• 32:34
  means as a data scientist, you have much more time to explore
• 32:39
  to, to kind of really think about how am I modeling this
• 32:42
  business problem? What is the best way to model
• 32:45
  it? What is the most accurate way to
• 32:46
  model it? And allows you to kind of to
• 32:48
  explore that space much more efficiently and faster than
• 32:51
  with, let's say, traditional feature engineering that takes
• 32:55
  so much time. Yeah.
• 32:57
  And I can't stop thinking about how two of the main feature
• 33:04
  stores have recently gotten bought or acquired for, you
• 33:08
  know, like Tecton got bought by Databricks recently.
• 33:11
  And then I just heard feature form got bought by Redis.
• 33:15
  And so it feels like that whole paradigm, was it maybe a dead
• 33:23
  end? If we can do things faster this
• 33:26
  way, why wouldn't we? That's a good question.
• 33:31
  I think this feature engineering approach to things, you know, it
• 33:37
  got revolutionized first in computer vision, then in natural
• 33:41
  language, but kind of it felt like that lets you know machine
• 33:44
  learning, predictive modeling was kind of stuck in the past.
• 33:46
  We were doing the same thing for 30 years, right?
• 33:50
  Like we were putting data in a single table and then, you know,
• 33:53
  we were training decision trees and then we were training
• 33:56
  support vector machines. And then I know logistic
• 33:59
  regression was fancy and then neural networks were fancy, but
• 34:02
  was all trained on the single table.
• 34:05
  And all these feature stores were built because because of
• 34:10
  that need to, to kind of compress the data or join the
• 34:15
  data into a single table. I think with this new wave of
• 34:19
  kind of bringing AI to machine learning, yeah, the need for
• 34:22
  that is drastically reduced. And, and we see it both in terms
• 34:27
  of productivity of building these models.
• 34:30
  We see it in building more accurate models and it just
• 34:34
  simplifies the stack, right? Like I, it's actually the, the
• 34:38
  statistics from industries is staggering.
• 34:40
  You need about two full time people per model.
• 34:46
  OK, right. So, so companies, you know, if
• 34:48
  they have 10 models in production, they have 20 people
• 34:52
  taking care of them. If you want 30 models, you need
• 34:57
  60 people. If you want 400 models, you need
• 35:01
  800 people. And that's that's the ratio.
• 35:03
  It's like 2 people, you need 2 full time employees to babysit a
• 35:07
  model and 30% of the model cost just goes to maintenance of
• 35:12
  running that model. Of course, then what also
• 35:15
  happens is, you know, people change jobs.
• 35:19
  So what happens is that in reality, you have all these
• 35:22
  models running in production that somebody built who's no
• 35:25
  longer with the company. Nobody wants to touch those
• 35:28
  models because as soon as they you don't even look at them,
• 35:31
  those models are going to break. Nobody knows what to do.
• 35:35
  Right. It's even worse than that.
• 35:36
  I remember back in the day, we had somebody come on here and
• 35:40
  they were working at Yandex and they said, yeah, my last six
• 35:46
  months there, I had to go and get rid of a bunch of zombie
• 35:52
  models that were just out there. Nobody wanted to touch him
• 35:55
  because nobody knew if they were making the company any money at
• 35:59
  all. And people were afraid that,
• 36:01
  well, if we take them offline and it did turn out to be one of
• 36:05
  those models that was making the company model or making the
• 36:09
  company money, that could be a really big problem.
• 36:14
  Exactly, exactly. That's, that's, I think there's
• 36:16
  maintenance and, and tracking of these models is a huge problem.
• 36:20
  But with this, let's say neural network technology, with this
• 36:23
  relational deep learning craft Transformers, it's, it's much,
• 36:27
  it's much easier to maintain. It's much easier to audit.
• 36:30
  It's much easier to to have them, to have them there to
• 36:34
  retrain them automatically and take care of them.
• 36:37
  So the maintenance cost becomes much, much less.
• 36:40
  Well, and Speaking of maintenance, if you're talking
• 36:42
  about the feature store, feature engineering way of doing things,
• 36:48
  a lot of the maintenance would just happen with the feature
• 36:51
  pipelines and that I know so many folks who had headaches
• 36:57
  because of that. And keeping the data fresh is
• 37:01
  not as simple as it sounds. Oh, it's, it's super hard,
• 37:05
  right? Like it's super hard.
• 37:07
  Actually, it's super, super hard, right?
• 37:08
  Even if you think about, you know, basically it means that
• 37:11
  for almost like if you have this counters this feature site for
• 37:14
  every event, you have to update your features.
• 37:17
  So whenever you make a transaction, your feature needs
• 37:19
  to be updated. If it's not updated, it's stale
• 37:22
  and it's all the information. And if you made a bit of mistake
• 37:25
  and updated it a bit too far in the future, then, you know, you
• 37:29
  are kind of having information leakage time travel and your
• 37:31
  models are, are are are, you know, good.
• 37:34
  So it's in reality a super big problem that I think people in
• 37:39
  research or academia, we don't really appreciate that because,
• 37:44
  you know, the data is always just given to us.
• 37:46
  Here's a training set, here's the test set.
• 37:48
  It's a fixed split. Who cares?
• 37:50
  You can kind of pre compute it all.
• 37:51
  But in reality, just yeah, updating those features is a
• 37:55
  huge pain. Yeah, well, this is fascinating.
• 37:59
  I mean, you've gone from computing features to just
• 38:05
  training these models. How many GPU's do I need to
• 38:07
  train a smaller model? Maybe take me through if it was
• 38:11
  fine-tuned to if I wanted to just build my own foundational
• 38:15
  model, what am I looking at as far as compute needs, costs, and
• 38:20
  then also time? Because if you're training a
• 38:23
  large language model, you need like a month, a month and a half
• 38:26
  type thing. It's actually interesting the
• 38:31
  the big surprise is how little resources you need.
• 38:37
  So it's actually very cheap, especially if you compare to
• 38:41
  large language models, then it's really, really cheap.
• 38:45
  Large language models are really humongous.
• 38:48
  You know, they kind of read the entire Internet and memorized it
• 38:53
  to do predictions. You don't need to do that.
• 38:57
  You, you need to learn how to recognize historical patterns
• 39:00
  and how they, how those predict the future.
• 39:03
  And that seems to require much less parameters.
• 39:06
  So these models, you know, you can do well with 50 million
• 39:09
  parameters, which is peanuts. You can do well with 100.
• 39:13
  And if you want, you can train a billion parameter model.
• 39:15
  But those are, those are super tiny when we think of LLMS. So
• 39:21
  what does this mean is you need less resources.
• 39:25
  Training times are in hours, not weeks or months.
• 39:30
  And very importantly, when you run things, when you put things
• 39:34
  in production, they are actually sustainable, right?
• 39:39
  LLM putting LLMS in production is very is is kind of many times
• 39:43
  unsustainable because it's too expensive.
• 39:45
  The each LLM call is just too expensive.
• 39:47
  But with these types of models, because they are smaller,
• 39:50
  actually they are quite efficient to run.
• 39:52
  So your compute cost is low and and that is then the benefit.
• 39:56
  It kind of, you know, the cost benefit analysis actually works
• 39:59
  out, works out well. So you don't need 10s of
• 40:03
  billions of parameters. These models can be actually
• 40:06
  quite, quite small from that point of view.
• 40:08
  And you can train them on a single GPU.
• 40:10
  Now, if you want the foundation model, maybe you need a couple,
• 40:13
  you need a couple of GPUs, but you know, not 10,000.
• 40:18
  Yeah, we don't need to build the data center to exactly, exactly
• 40:21
  train this model. One thing I wanted to say is
• 40:25
  that this structured data foundation models, as we said
• 40:30
  are smaller than large language models, easier to train, have
• 40:36
  less parameters and this also means they are cheaper, cheaper
• 40:40
  to operate. One thing that is maybe
• 40:43
  fascinating here to me is that you can actually build a pre
• 40:47
  trained model that allows you that allows you to kind of
• 40:51
  answer tasks ad hoc. You don't have to even fine
• 40:54
  tune, you don't have to kind of train for a specific prediction
• 40:57
  task, but you can almost give a set of training examples in
• 41:00
  context to the model and the model is going into a single to
• 41:04
  forward pass make you an accurate prediction if that
• 41:08
  makes sense. And you interface with these
• 41:11
  models through prompts like you do with large language models.
• 41:14
  That's a great question. So if you have a pre trained
• 41:17
  model, the way you interface with it is scan is through some
• 41:21
  kind of prompting. It's a domain specific
• 41:24
  structured language we call predictive query and it
• 41:28
  basically has two parts. It has a predict part and the
• 41:31
  four part predict says I want to predict this this quantity for
• 41:36
  this specific entity. So we say I want to predict your
• 41:40
  purchases next week. I want to predict some of your
• 41:43
  transaction values next month. I want to predict probability of
• 41:48
  fraud for this transaction. So you specify that and yeah,
• 41:53
  out of this is basically almost like a prompt.
• 41:56
  And based on this, the model is then making a making a
• 41:59
  prediction. If you are talking about a
• 42:01
  general foundation model, if you find you then the model just
• 42:06
  takes the data and gives you the prediction you want.
• 42:08
  So it's not like I'm using Sequel or Python.
• 42:11
  It's not. It's not like to prompt the
• 42:14
  model you are using. Almost like Sequel where you
• 42:18
  specify what's the quantity you you want to be predicted.
• 42:23
  Yeah, it's like. Pseudo sequel prompting this
• 42:25
  pseudo sequel? Sure, it's a sequel that selects
• 42:27
  over the future. That hasn't yet happened, so you
• 42:30
  need to predict that future. That's nice.
• 42:34
  And you said something in a talk you gave that I wanted to dive
• 42:38
  into more, which I found fascinating, and it was agents
• 42:41
  need predictive AI tools. Can you break that down for me?
• 42:46
  Maybe to start at the beginning right?
• 42:48
  Like why are we making predictions, right?
• 42:51
  We are making predictions because we are making decisions
• 42:55
  based on those predictions, right?
• 42:58
  Why am I predicting fraud probability?
• 43:00
  Because based on that fraud probability, I decide whether to
• 43:03
  stop a transaction or not. You know, why am I predicting
• 43:07
  churn probabilities? Because based on that estimate
• 43:10
  of churn and then taking some action to bring that customer
• 43:13
  back, right? Why in a hospital, I'm
• 43:16
  predicting surgery, readmission probability, because based on
• 43:20
  that I decide whether to discharge the patient or not,
• 43:23
  right? So I'm or you know, in, in, if
• 43:26
  you think in finance, you could say, OK, why am I, why?
• 43:28
  Why am I predicting your probability to default on a
• 43:31
  loan? Because then I decide, do I give
• 43:33
  you the loan or not, right. So prediction is really a basis
• 43:37
  for decision making. So if now we we believe in
• 43:41
  autonomous agents, autonomous agents need to make decisions,
• 43:47
  right? And you don't make decisions
• 43:49
  based on common sense, right? It's OK, it's not terribly
• 43:53
  wrong, but it's not optimal as well.
• 43:54
  You want to make decisions that are rooted in the data.
• 43:57
  To make decisions that are rooted in the data, you need to
• 44:00
  kind of predict the outcome of those decisions, right?
• 44:03
  And right now, what's the bottleneck in, let's say,
• 44:06
  deploying this type of autonomous agents is their
• 44:09
  ability to make decisions. And this means that now you
• 44:13
  have, you have your beautiful LLM agent, super, super
• 44:16
  intelligent. And then, you know, you have to
• 44:18
  build manually some machine learning model that's going to
• 44:20
  say, oh, what's the probability of churn, right?
• 44:24
  Because imagine a simple agent that says, let's go identify
• 44:28
  people that are most likely to churn.
• 44:29
  Let's identify the best offer to, to send to, to each of those
• 44:34
  people. Let's write a nice personalized
• 44:36
  e-mail and send it out. Right.
• 44:37
  In this workflow, there are two big decision problems which
• 44:42
  people are likely to churn. What is the best offer to give?
• 44:46
  And those are two predictive tools that you need to solve
• 44:50
  this use case end to end. Damn, I see where you're going
• 44:56
  with this that is. Agents will need predictions.
• 45:00
  Maybe we have not realized that yet, because what agents do
• 45:03
  today is they query some document database and retrieve a
• 45:06
  couple of passages, but those are not, those are not real
• 45:10
  agents, right? The real agent is actually
• 45:12
  something that does something to you, right?
• 45:14
  Like or imagine I'm a customer support agent, I want to
• 45:18
  estimate what's your lifetime value, what's your probability
• 45:22
  of churn? Because based on this, I will
• 45:24
  talk to you very differently. I will give you different
• 45:27
  offers, I may suggest different remedies.
• 45:29
  Those are predictive problems, right?
• 45:32
  So how can we leverage the predictive models that we've
• 45:35
  been doing, but like you said, to make sure that they're
• 45:40
  faster? Exactly.
• 45:42
  I think the, the benefit is in basically using relational
• 45:46
  foundation models that the, that these LLM agents can, can use as
• 45:51
  tools to credit for these predictions.
• 45:53
  So whenever you show up, they're like, how likely this person to
• 45:56
  turn, how you know, what is their lifetime value?
• 45:59
  Oh, now that they have these estimates, this is what I'm
• 46:02
  going to do, right? What's the next best offer to
• 46:05
  give to this person? I'm, I'm, you know, I'm making
• 46:08
  this decision on the fly based on a predictive model that is
• 46:11
  rooted in my data that gives me an accurate prediction.
• 46:14
  Rather than use some kind of common sense hallucination to
• 46:18
  say, oh, what should we do here? Yeah.
• 46:21
  Or rather than how it's been done right now, which is that
• 46:26
  we're creating these predictive models and maybe there's some
• 46:32
  advanced teams that are offering up these predictions to the
• 46:35
  agents. But as we just have been talking
• 46:39
  about for the last 40 minutes, that is a cumbersome process the
• 46:43
  way that we're doing it right now.
• 46:45
  Exactly, that's the alternative. It doesn't scale.
• 46:47
  It means you have to hire 2 humans for every predictive
• 46:51
  problem you have. And that's that's a lot of
• 46:54
  humans to hire. It's so obvious, like, why?
• 46:58
  Why don't we have that right now?
• 46:59
  Or maybe you've seen some teams that are actually doing this
• 47:03
  because like you said, you know, the step A, step B, step C is a
• 47:07
  super common use case and it's a pattern that happens within
• 47:12
  companies every day. Can we, have you seen agents
• 47:17
  being used like that? If not, yeah, like what is the
• 47:22
  blocker? Yeah, I would say we have seen
• 47:25
  agents being used like that and we have a couple of clients
• 47:31
  deployments where where this is happening.
• 47:34
  I, I think the, the, the blocker right now as I see is the, is
• 47:39
  the, is the knowledge, right. It feels like there is this AI
• 47:45
  delirium out there and then there are all these kind of
• 47:49
  quiet, quiet people who are doing real work and they, you
• 47:54
  know, they are not being being heard.
• 47:56
  And that seems to be kind of a big a big a big disconnect to
• 48:00
  me. Is there anything that I didn't
• 48:02
  ask that you want to talk about? That yeah, there is, there is
• 48:06
  actually hope for machine learning as well that right now
• 48:09
  is a bit on a, on a, on a side track or people, don't they,
• 48:12
  people don't even realize or we seem to have forgotten how
• 48:17
  valuable structured data is because it's kind of the ground
• 48:20
  truth of the business. And it's it, it, it captures the
• 48:25
  blueprint the of, of the business.
• 48:27
  And we seem to have forgotten all that.
• 48:29
  And you know, we are now excited about answering questions about
• 48:32
  documents. But I think the, you know, the,
• 48:36
  the world is going to come back because enterprises are here to
• 48:40
  make, to make accurate decisions, to make business
• 48:43
  impact and to drive the business forward.
• 48:45
  And for that you need prediction.