Latest / Agentic Conversations (formally mlops.community) / Real-Time Forecasting Faceoff: Time Series vs. DNNs // Josh Xi // #305

Transcript

Real-Time Forecasting Faceoff: Time Series vs. DNNs // Josh Xi // #305 · Agentic Conversations (formally mlops.community)

0:00
Josh XI was lift for a little bit over five years.
0:04
Staff data scientist. How do I want to take my coffee?
0:09
OK latte is good. I like it creamy milk.
0:13
I like I found recently found those 6% fat milk that's super
0:18
creamy, almost like half half, but it tastes just so good.
0:24
Well, hello, hello everyone. We are back for another
0:29
Envelopes community podcast. I'm your host, Demetrios, and
0:32
today we talk about time series, machine learning models versus
0:37
deep neural networks. And Josh has some strong
0:41
opinions on them, but we would love to hear if you also are
0:46
opinionated about one or the other.
0:48
He just told me as soon as we stopped recording that he would
0:51
love to get comments in case he's missing something and he
0:55
wants to know how and what folks are doing if they are doing it
1:02
differently than him. Let's get into this
1:04
conversation. First off, tell me what you're
1:12
working on because I I find it fascinating and it is deep, deep
1:17
in the weeds and I love it. So I want to know everything
1:20
about it. So I'm in this big or core
1:24
marketplace and that's basically the team managed to supply
1:29
balance of the lifted marketplace.
1:31
So essentially we are platform teams.
1:33
So you have the demand side, the supply side and you want to
1:36
achieve the market balance between the two.
1:40
So there's different levers you can do that like pricing.
1:43
Typically if the demand is too high, you can be surprised to
1:46
suppress the demand or you can offer driver incentives to sort
1:51
of acquire more drivers. So both can happen in either
1:55
real time or long term. So long term pricing is more
1:57
like coupons. You can send out coupons so it
1:59
can attract more riders onto platform.
2:02
But all of these models or levers, they take lots of the
2:05
signals that goes into them. So like the most basic one will
2:10
be forecasted demand or forecasted supply.
2:13
So my team is called market signal.
2:15
So that basically provide all these key signals or features
2:21
that goes into these models. So you need to know what's
2:25
happening now, but you also need to know what's going to happen
2:27
in future. So there's lots of forecasting
2:29
for those letters. And I think I've heard something
2:34
from folks who left Lift about just the sheer amount of models
2:41
that you're running at any given time.
2:43
And maybe it's not that it's different models, it's just
2:48
different models in different parts of the world or parts of
2:51
the US or. And so it's it's very similar
2:54
models, but kind of different, I guess.
2:57
Same, same, but different. Yeah.
2:59
So it's a, it's actually a huge amount of models, so basically
3:03
lifted very data-driven. So all of these like levers
3:06
there running in different regions.
3:09
So at the same time. So like in so my team 11 set,
3:13
the problem we are facing is basically we call them real time
3:16
forecasting. So basically for every Geohash,
3:22
for those who are not aware of Geohash, like it's like a
3:26
typically a standard like zone definition, kind of like zip
3:30
code, but they had different levels.
3:31
Like we're usually looking at Geohash 6, which is like 1 by 1
3:36
mile ish, like a cell in a location.
3:40
So every city you typically have like a couple thousand sometime
3:43
to even like a 10,000 geohashes. And so basic for every geohash,
3:49
we need to generate a forecast in real time for the next say 5
3:54
minutes up to an hour. So you will probably have
3:56
millions of data points to forecast at the same time.
4:01
Wow, OK, so each geohash which makes sense because you want
4:06
something that is accurate down to the block.
4:09
Yeah. So typically if you're thinking
4:11
about like pricing or like driving incentive, like
4:16
essentially you're trying to reallocate drivers in a way or
4:19
attract them to certain busy area.
4:21
So it needs to have certain like a granularity.
4:25
So if you're just forecast the whole region, it doesn't really
4:28
help the drivers to to to know like where they should go.
4:31
So in in order to have the actual action you can take to
4:38
make some sort of impact, you have to sort of have that
4:41
granularity. So for that reason, we typically
4:44
start from Geohash 6 level, but depends on use cases.
4:49
Some of them we might aggregate them to a high level to sort of
4:53
fits that those needs. Cause the bigger the area they
4:57
are, the more density you have and potentially the more
5:00
accuracy you can achieve. And how often are you using
5:04
features or data outside of the platform?
5:08
That is something like there's the Super Bowl going on and
5:13
there's probably going to be a lot of demand around the stadium
5:17
when the game is about to start, or from 2 hours before the game
5:21
and then two hours after the game.
5:25
That's a really good question actually.
5:26
So we try our best to sort of getting external sources events
5:32
is definitely one of them. So we do working with some data
5:37
contractor to get some input. So events, airport information
5:43
is like flight, landing, departure, anything that we can,
5:47
we think that could be helpful, but in reality there's lots of
5:50
challenges just to ingest those data.
5:52
So taking events as example, like if we can easily, typically
5:58
we have information about like the event start time, like which
6:04
of the ticket time when people normally will go to the event.
6:08
But the tricky part is the event and time.
6:12
Like imagine you're sitting in the football game and it's like
6:16
2 minutes left on the clock. And that two-minute can sometime
6:19
mean just two minutes or sometime that can mean half an
6:22
hour. If there's overtime or any sort
6:26
of uncertainty depends on like how close the two game, the
6:30
current score is between the two teams, right?
6:32
So there's for all the uncertainty for that.
6:34
For all the international listeners, that is when American
6:39
football 2 minutes is not actually 2 minutes for football
6:44
around the rest of the globe, soccer, right?
6:47
Yeah, I guess you have a little bit of leeway at the end of the
6:50
game, but not like in American football.
6:52
But anyway, sorry, I didn't mean to cut you off.
6:54
I just. Wanted to Oh yeah, that that's
6:56
totally fine. Yeah, soccer.
6:58
But yeah, for soccer sometime I go into overtime, so that might
7:02
be means another half an hour or longer.
7:06
So yeah. So we've been having trouble
7:08
trying to figure out what's the rights like event and time.
7:13
That's definitely a big challenge.
7:16
Other thing like we mentioned about weather data.
7:20
So we also have been doing lots of sort of analysis of how
7:24
weather impact the the demand essentially.
7:29
Well, I think most people their first instinct is like bad
7:33
weather, more demand, which is maybe generally true.
7:38
But when we are really looking at the data like looking at
7:42
creating all these like precipitation features,
7:46
temperature features or do all kind of combination on top of
7:49
them and trying to find correlation with our demand
7:52
data, it's actually lower than what we expect.
7:55
No way. Want the conjecture here is
8:00
actually sometimes not really about what the temperature is or
8:03
how much rain you have. It's more about like people's
8:07
expectation of what's going to happen and what really happened.
8:11
So the forecast stays, OK, it's going to rain and it's pretty
8:15
accurate. Everybody is more likely
8:17
mentally prepared or they already have some way to sort of
8:22
prepare for the big ring heavy rain.
8:25
But if something caught them off guard, everyone's like, OK, let
8:30
me call a taxi. So or of course on Uber, Lyft.
8:33
So I think that makes a huge difference.
8:37
And also snow, one of the in most interesting finding that
8:40
was like back in 2014, that's when I actually first start to
8:44
look into taxi data versus the weather and like everyone's like
8:50
OK, snow is going to make huge difference on taxi demand.
8:54
So we were look, in the first half of the winter, we were
8:57
looking at the data, we saw some correlation, but somehow
9:00
second-half of the amount of snow has really poor correlation
9:03
with the taxi demand. And you know what happened?
9:07
Well, that year was like a super bad winter.
9:10
And in the second-half the season, most of the city start
9:15
running out of like their salt to clean up the streets.
9:20
So they don't clean the streets. Nobody can travel.
9:24
People just give up travel or they just start decide to stay
9:27
at home. So there's really not much
9:29
collation between snow precipitation versus like how
9:33
much demand you will have. So those are external
9:35
information. Usually it's very hard to put
9:37
into the model. So yeah, that that's, that will
9:41
definitely, those kind of factors definitely affect how we
9:45
boot out forecasting models or if like such a model we're going
9:50
to work better than others. I thought you were going to say
9:53
when it rains people just stay home.
9:56
So the demand is lower, in which I can imagine a certain subset
10:00
of folks stay home. But it is more like what you
10:03
were saying. If it says it's about to rain,
10:06
then I am more mentally prepared to go out in my rain jacket or
10:12
with my umbrella. But if it is supposed to be
10:15
sunny and then it starts raining, it's like, Oh no, I
10:18
didn't bring my umbrella. I didn't bring the.
10:20
I'm not prepared for this. I need to get home or get to
10:25
wherever I need to be quick. Yeah, that, that's totally true.
10:29
And also I wonder like if pandemic changed people's like
10:32
travel behavior. So like before, that's like
10:38
everybody's like sort of mandatory to go to office.
10:42
So even if it's raining, they're more or less likely to sort of
10:45
OK, it's raining. I still need to go to office
10:47
these days, Say, OK, if this rains, I'm in a bad mood.
10:51
You know, I only need to go into office two days a week.
10:54
Let me pick a different day or something.
10:56
Yeah, yeah, completely. It, it changes how likely we are
11:00
to do something. When we talk about this data
11:03
acquisition though, that just seems like a mess, man.
11:07
It seems like something that is so difficult a, to get the right
11:12
data and then B, to clean it or to transform it and then create
11:17
insights from it. Especially for this external
11:20
data. I imagine you have some kind of
11:22
pipelines set up and you're probably constantly tweaking
11:27
them or you're doing, you're playing around with it to see
11:30
can we create better features from this or something that
11:33
gives us more insight that we can feed to the model.
11:35
And I off base there when I say that.
11:38
Yeah, yeah. That, that's definitely
11:40
something we do a lot internally.
11:43
We have teams like looking at the event data to curate them.
11:48
But in the end, it's like so much like labor to just get them
11:53
right. We focus a lot on the top events
11:56
and that's the one that actually works well in the some of the
12:01
our models. But for most of them it doesn't.
12:05
So that's sort of kind of lead to what I want to sort of
12:09
thinking about him. I had to talk today is like why
12:14
traditional like time series broadcasting works better in
12:18
reality? Like the end, a lot of people's
12:21
concepts like, OK, the model is going to learn the feature on
12:24
its own. So we can plug in as many data
12:26
as you want. So let the model learn what's
12:29
important, what's not. That usually means lots of
12:32
training data you will need and lots of computation power you
12:35
need versus time series type of model like auto regression.
12:40
ARIMA is trying to predict basically the future by mostly
12:44
folks almost exclusive focusing on the history because at least
12:50
even the market is not like even the market is not stable.
12:54
They are not sort of keep having the same trend, but within a
13:00
short amount of time, things tend to repeat on its own time
13:03
series models. Looking at what happened
13:06
yesterday, the day before, same time last week, it actually
13:11
already covered lots of what's going to happen in your future.
13:15
So this type of models is something we actually been sort
13:21
of it has been our top choice so far.
13:24
And because they're sort of very interpretable because your
13:28
future is some sort of a waiting average of your history.
13:33
So based on how much you believe the feature going to change
13:38
relatively to the history, you can relatively, I would say
13:43
easily to apply any human intervention to do some sort of
13:48
adjustment, say based on what we know, OK, last week there's no
13:53
Super Bowl, but you know, next week can there will be a Super
13:56
Bowl. So you can actually easily to
13:59
adjust or demand in that case. But with DN, I guess in, in the
14:06
case of Super Bowl, we can't do the same listing if you know, DN
14:09
is never trained on Super Bowl. So you can make adjustment too.
14:13
But there's also all all other kinds of local events that's
14:17
happening and some of them might be much harder to make
14:20
adjustment and you're not sure if DN capture that or not when
14:24
feeding those models. So you have to versus auto
14:28
regression. You know, you sort of know for
14:30
sure. It's like, OK, if I don't
14:34
because typically like auto regression does not, it's just
14:38
hits revolt values. So it does not really explicitly
14:41
put any events information there.
14:44
So you can just assume there's no events like in my model.
14:50
So you can't easily add on certain things or there's more
14:55
advanced models like decomposition models, you can
15:00
decompose trend seasonality. And so you sort of know like,
15:04
OK, which part I need to adjust? If I believe there's a local
15:08
events happening, it's a spike. It's it's not in the
15:11
seasonality, it's not in the trend.
15:13
So you can take those two components, add on your spike.
15:17
So that make it a bit much easier to adjust the model,
15:21
adjust the forecast and make it a little bit more accurate.
15:25
And it almost seems like with those spikes, you can also
15:28
leverage the history to say, OK, this looks like a spike that we
15:32
had maybe 2 weeks ago or last week.
15:35
So we'll trend in that direction.
15:39
What I'm really interested in there's, there's kind of two big
15:42
questions that I have with DN, NS.
15:45
And then also, I've heard a lot about time series foundational
15:51
models. And so the First off, I guess
15:55
the simple question is have you played around with time series
15:57
foundational models? Have you had any success with
16:00
them? Because I've generally heard
16:03
from people in the community that they are not very valuable
16:07
and they haven't had success with them.
16:09
So actually I, I haven't, I played well, I actually play a
16:13
little bit with time, chat time, GBT.
16:16
I know my coworkers also have play a lot with some of them.
16:20
So I, I actually learned my first hand information from
16:23
them, not myself, but my learnings based on my
16:28
conversation with them is it's accurate when you're looking at
16:36
some of our use cases because we have actually have forecasting
16:38
for most rear tying spatial temporal models like what I
16:42
described earlier that like super granular in that sense.
16:45
And because every minute you're looking at forecast next 5
16:48
minutes, 10 minutes up to an hour and it's all these
16:53
thousands of cells. So there's lots of variances,
16:56
all kind of things going on. And you also need it to be fast
17:00
because every minute you just keep refreshing forecast versus
17:03
there's another type of forecasting which is more like
17:06
offline week or short term. Near term, every company or team
17:12
might use different names, but they are looking more at like
17:14
say a regional like the total host San Francisco or break up
17:18
into a few sub regions and looking at like hourly daily
17:24
signal values for the upcoming week or two.
17:27
So most the learning actually is it works well for those type of
17:34
models, works well for the later use cases.
17:37
The first use cases haven't been that well so far.
17:44
I think this, that's mostly talking about accuracy, but I
17:49
think beyond accuracy, another really big issue is the real
17:53
time like Cadence. Yeah, it's just happening so
17:56
fast and those type of models are a little bit too big to sort
18:00
of spin up quickly and just keep emitting features.
18:05
And I'm not sure if there's already a sort of engine for a
18:08
designed to leverage those models for use cases like that.
18:14
And especially at that rate that you need, it makes a ton of
18:19
sense. And that was the question that I
18:22
wanted to talk about with the DN NS is what kind of
18:25
infrastructure is needed to serve those types of models or
18:30
trying to utilize those models in your use cases?
18:34
And is that not adding extra headache to the problem when
18:41
later you find out like, damn, these don't even perform that
18:44
much better. Why are we breaking our back to
18:47
support the DN NS? It's just because maybe some
18:51
folks want to have it on their resume.
18:53
Is it like, what is it about? I, I know it's not that you, you
18:57
all probably wanted to like thoroughly test it.
19:00
But at the end of the day, if it's much harder to support and
19:04
they're not giving you that much of A lift, then I'm assuming
19:08
that it's much harder to support.
19:11
I would love to hear from you what in reality the difference
19:15
is between supporting these two models.
19:17
Let's compare 2 type of models, which actually is something
19:20
we've been testing a lot on. And one is sort of auto
19:23
regression, which I mentioned earlier will features weighting
19:28
average of your history. And the other one, let's see.
19:34
And this is there's this very famous paper from Microsoft Lab
19:40
focusing on spatial temporal DNS.
19:43
There's also other models like LTSM use board time series
19:46
forecasting too. Although most LTSM papers focus
19:51
on just one single time series. In our case we're talking about
19:54
many like thousands of cells in the spatial like in the city or
19:58
something. So you have also have to capture
20:00
the spatial correlation there. So make the model a bit more
20:03
complicated. So the Microsoft paper is 1 one
20:07
of the most well studied or used paper.
20:12
In in lots of the application or research.
20:17
So from training I think the two make a huge difference.
20:21
First is auto regression, do not use GPU just CPU, a single CPU
20:30
if you want to do any back testing training, it's fast and
20:35
a spatial DN you use GPU and so tip cost wise it can be at least
20:40
100 times difference. That's from our first hand
20:43
experience now serving them. So if you want to just like sort
20:51
of like DM model, we do not necessarily need to use GPU
20:55
because your model weights already pre calculated.
20:58
So you can just spin up machine and just take the weights,
21:04
reconstruct a model and take your inputs a little bit more
21:08
computation. But I wouldn't say it cost that
21:11
much extra because thinking about it typically in our case,
21:16
we do forecast like the cadence like every minute.
21:19
So it's every minute you try and predict for the next say 30
21:23
minutes. So as long as you can't finish
21:25
your whole forecast in like 30 seconds, it should be good
21:29
enough to go. So I think just a general
21:32
machine loading the model weights, reconstruct the model,
21:36
take your input through the calculation, 30 seconds is
21:38
sufficient time series models much less weights, so so of
21:44
course much faster. So student definitely no issue.
21:48
So there's why there's not much sort of difference from cost
21:53
perspective on the inference part.
21:57
How about on the data side when you're training it?
22:00
What kind of data you need for each of these?
22:02
Because I feel like you need a lot of features for.
22:05
The. Auto regression, but maybe you
22:08
don't need the features or you don't need it as clearly in the
22:11
deep neural network. Oh yeah, that's a good point.
22:16
That's another differences on the training site is the
22:20
training data. So for ARIMA we can just, you
22:24
know, take a couple weeks of data, you know, experience and
22:29
video like features, which is future values versus historical
22:33
values. If if you're trying to use
22:35
history for like saying time a week from 1:00 to 2-3 weeks ago,
22:41
then you also need to much a little bit longer history.
22:44
For DNN it's doable with small amount of data.
22:51
Then the question is like the, the weights might not be as good
22:54
in the beginning because you have more weights, you are more
22:58
likely to like run into it like on like on the training
23:02
situation. So ideally you want to take much
23:04
longer history and auto regression.
23:08
You can't even start without actually a sort of any weight.
23:12
So you can just assuming a moving average.
23:14
It was like 20% on last three minutes and 20% same time of day
23:20
from yesterday, 20% from the David sorry the week before or
23:24
something. Just making up your own prior of
23:27
how the weight the average will be.
23:30
What you can do is the actions online, back to the online
23:34
difference, although there's no cost, there's actually a huge
23:38
sort of advantage with auto regression or classic models is
23:43
you can refit them. Those are linear models.
23:46
You can refit them at the Super fast because it's just a few
23:50
weights, right? There's probably 20-30
23:53
historical features you take average on.
23:56
So we can't just you have a new observation from the past 5
24:00
minutes or 10 minutes, half an hour, which depends on what's
24:04
your retraining cadence you want to do.
24:08
And you can just plug into this linear model and just run the
24:12
refit adjust or wait. So we can do online learning,
24:15
make the model sort of adaptive to whatever it's changing in the
24:19
real time marketplace. So any sort of change it like a
24:22
spikes picking up, you can use refit a model to quickly put
24:26
more weights on maybe your recent values to catch up the
24:29
spikes. With DN, it's a much larger
24:33
model. So you can do retraining too,
24:36
right? Because that's how we train the
24:38
model anyway. It's like a batch training or
24:40
something. You can always take a small
24:41
sample data and retrain the model.
24:43
But the problem comes in to the cost cause retraining in ADN
24:49
model is actually very expensive too.
24:51
So similar to the training case, anything you want to do
24:54
retraining online, trying to adapt to any changing, changing
24:58
situations happening in the marketplace that will incur much
25:03
higher cost. So that's actually another sort
25:07
of shortcoming of DN in practice because that cost you can
25:12
probably you can only do less often retraining, which also
25:16
means you are sacrificing your accuracy with with less retrain.
25:22
So yeah, it's and it's similar experience that can also means
25:27
10 times or 100 times differences in your training
25:30
cost. Well, especially if, like you're
25:34
saying, you're retraining continuously and constantly just
25:38
retraining when you're learning new things about the world.
25:41
So I imagine, like, how often are you retraining?
25:45
Is that pipeline just set up to be triggered every couple
25:49
seconds? Are we talking every couple
25:51
hours every day? Every minutes right now like
25:54
it's, it's just a choice because the machine is basically standby
25:59
for our models the whole time. So I know like inside the
26:05
marketplace team, my team has one of those expensive team to
26:09
like using machine powers. Yeah, yeah.
26:13
But it's worth it I guess if you're continuously getting that
26:16
accuracy. Just that .3% accuracy lift is
26:22
gaining you a whole lot of revenue, so you it makes sense
26:26
that you want it to be as accurate as possible.
26:30
Yeah. We, we do some sort of a cost
26:33
benefit trade off more or less. So how much we've been running
26:38
these machines, how much they cost versus like the estimated
26:42
impact of the accuracy. So those are trade off in our
26:47
daily work we need to look at too.
26:49
Yeah. And so I imagine it's got to be
26:52
automatically re triggered for this retraining.
26:55
How do you then go and add extra value?
26:59
Is it that you introduce a completely new model and then
27:03
that gets thrown into this retraining loop?
27:07
Or is it that you just start are continuously updating the model
27:11
and adding new data sources? Like where do you plug in to
27:15
make sure that whatever model is out there is performing the best
27:19
possible? With online auto regression
27:22
models, it's has been working well for most of the use cases
27:27
like capturing that spike and making it adjust faster to the
27:36
marketplace. The challenges so far we have
27:40
learned is actually probably mostly the wrong events.
27:44
Our models do capture that like when events start to like
27:48
increase demand, we our model will capture that.
27:54
But there's a little bit delay in our model here.
27:58
So depends on how we set the learning rate in our model
28:03
relatively to the historical data.
28:05
It's basically you have a old weights, you have learning rate
28:10
based on what you observe, it's battling OK.
28:12
Should I put more weights on my recent spike or should I just
28:15
trust more in my history? So it has been very hard to
28:20
tuning those parameters because every region is different.
28:24
So every region have you typically have their own models,
28:26
but we haven't really have a goods approach to finding that
28:31
perfect parameter tuning for each models.
28:33
Ideally you can just keep doing everything offline, keep testing
28:38
and running lots of machines let that take cost lots of money.
28:44
So we have some sweet range. Typically this like parameters
28:48
works well. So we use them for all the
28:51
models we are running. So on top of that, what we can
28:55
do is you can always observe how the model perform in the last
29:01
few minutes, right? Once we see the real data of
29:04
what to forecast it and what's the error is.
29:08
So for external adjustment, instead of using sort of OK, you
29:15
can't instead of like having people looking at the data and
29:19
making adjustment or say there's a football here.
29:23
So let me add more demand. Another approach actually just
29:29
looking at the forecast errors and see how constant or that
29:33
error has been. And you can add some sort of
29:36
bias correction or some sort of adjustment by setting up some
29:41
some sort of a heuristic rules to OK, if it's say last 10
29:46
minutes, it's constantly on the forecast.
29:48
Let's bump up the demand. So the the adjustment can be
29:54
like using ratio based on like how much it was like if the
29:59
forecast has been constantly 90% ish of what the actual is, you
30:03
can sort of divide by 0.9 or if the so.
30:09
But if you're thinking about like actual human intervention,
30:13
so far we haven't really tested out because the type of problems
30:18
we are facing is like spatially like huge, right.
30:23
You have down in the sales in the region, it's really hard to
30:27
just looking at each one of them.
30:30
So everything is sort of automated based on more like
30:32
this accuracy versus another sort of direction we're looking
30:37
to is actually more ensemble models.
30:39
So auto regression is just one of the model.
30:44
It's easy to explain. So that's what we picked, but
30:46
there's other sort of linear in the linear model area, you can't
30:50
run refit quickly or you can also like basically changing the
30:56
model set up based on your assumption of how data
30:59
especially distributed. You can basically apply more new
31:02
models to sort of running them at the same time as long as
31:08
there's relatively small you you don't it doesn't really cost
31:11
that much and you have different models running at the same time
31:15
and you have different performances.
31:18
And then based on the recent performances, you put more
31:21
weight on the one that give you better forecast.
31:25
So that's that's another approach to sort of a make bring
31:29
up the accuracy. So it's not like you're taking
31:32
the average of the five models that you're running.
31:35
It is that you're taking the model that is being the most
31:39
accurate. Yeah, you can also weight them.
31:42
So the the best one give 80% the weight on the forecast and the
31:49
second best give them 20%. So it's like a weighted average
31:53
of like the two best models. That's another sort of approach
31:58
we are taking. So, so far we, I will say my
32:02
team have been focusing mostly on mostly on using those
32:05
approach to get better accuracy. Yeah, 'cause just humane to
32:12
mention. So we only have a few data
32:14
scientists on the team. It's hard to just like, OK,
32:16
let's look at what's happening here today, what's happening
32:20
there tomorrow. That that's that's a little bit
32:26
like beyond what we can't do right now.
32:29
Yeah. And it does seem that the idea
32:32
of the more quote UN quote simple models versus the deep
32:36
neural networks are easier to interpret.
32:42
So, and actually another way we can actually do, which is
32:47
actually sort of what we hoping to achieve this year is to
32:53
online models like very granular.
32:56
But we also I also mentioned about like another type of
33:00
forecasting, which is people running at the regional level
33:02
forecasting or sub regional. Like instead of looking at the
33:07
cells, they look at like maybe 10 sub region in the city and
33:12
they looking at hourly forecast for the next few days.
33:16
And so you from those models you can get the trend to
33:20
seasonality. And they also typically those
33:22
model also they have people more people to look at the impact of
33:27
the like event. So they will do like sort of
33:31
some sort of a new adjustment based on our knowledge of OK
33:34
around event time. This is how much spike you will
33:37
see. And so we are sort of thinking
33:40
taking that as an output and then feed into actually the last
33:45
layer of our real time forecasting.
33:47
And sort of once we forecast, right, we have say we are
33:52
looking at a cell level. And for this event happening in
33:56
the sub region, we can aggregate all the cells and see what the
33:59
total demand we forecast it and see how far that is from the
34:03
offline forecast. And then you can take that
34:06
difference and do a multiplier or something.
34:10
And that's how we can do adjustment.
34:13
Yeah, the I think it's more and challenge because we have to
34:16
have the retire system talking to the offline system.
34:21
But the concept is pretty straightforward.
34:24
Taking output of the retime forecasting, compare that to the
34:29
offline forecasting with human intervention and just check the
34:33
differences between the two. And so is the idea there that
34:39
you know what it should be because of the seasonality or
34:43
the you know what it should be because last year at this time
34:47
it was this or, and you're looking at a, you're opening the
34:51
aperture. And by opening the aperture,
34:55
it's going to give you another feature that you can feed into
35:00
the model. Yeah, 'cause auto regression
35:05
typically look at the loss what happens last week and sort of
35:09
capturing the average sort of season seasonality based on last
35:13
week or two weeks of data. So if you offline, offline has
35:19
more longer, they look at longer histories and they do with
35:24
events more explicitly. So if they believe the event's
35:29
going to bump up the demand and they already finished like did
35:33
their adjustment and their final outcome.
35:36
So our assumption here is they have a better capture of those
35:40
like special events or seasonality.
35:43
Because they've seen it before last year or last quarter or
35:48
whatever, and I get it. Yeah, yeah.
35:51
And offline models, typically they do training looking at two
35:55
years of data 'cause it's typically a few time series or
36:00
and over like just a few sub regions.
36:06
So it's not really that much data to to sort of train using a
36:10
longer history. So it's totally doable.
36:12
And also for their use cases, typically they run on a weekly
36:17
or daily basis. They run forecast once every
36:20
week looking at the daily or hourly forecast folder upcoming
36:24
Week 2. So they don't have to just like
36:27
in real time keep generating those forecasts.
36:29
So cost to them is a less concern.
36:32
So they have more their model tend to be a bit more
36:35
complicated dealing with seasonality, external events and
36:42
they also looking at regional level.
36:43
So and they, they can also sort of pick out the major events in
36:50
the region and sort of to do it just on on top of that.
36:54
So they definitely have more a little bit easier in terms of
37:01
making adjustments, beating more complicated models.
37:06
Because they've seen it. Yeah, that makes sense.
37:08
It's like it's not so new and it's not like just out of left
37:12
field and you have that data. I I didn't realize that it you
37:16
get 2 years of data. Yeah, or even more if you want
37:21
to. Yeah.
37:22
So if there's a a local sports team and they have during their
37:27
football season, they have a game every Sunday.
37:31
And then, you know, all right, when it is football season, this
37:34
is more or less what the demand is going to be like because
37:37
we've seen it for the last two or more years.
37:40
Yeah. And for those, and it's sort of
37:44
more or less embedded in the future on the like the time
37:49
series already. And so sometimes it's more like
37:53
the the thing that happened only like a few times a year and
37:56
that's very hard for the time series to pick up.
37:59
So in that case, like likely human intervention will be
38:04
needed. So I think for whoever working
38:07
on long term forecasting they they the last step they focus a
38:11
lot on those like big but only occur at a couple times a year
38:16
event. Yeah, I can see that.
38:19
So the other thing that I wanted to ask is when you are testing
38:24
out these new models, do you do like side by side analysis?
38:30
Are you doing some kind of champion Challenger release?
38:34
Is it? I think there was something else
38:37
that I had heard of, but I can't remember how it worked where
38:40
you're just giving it dummy data and you're simulating what it
38:44
would predict and you're seeing if it would be more accurate
38:48
than what you have live. Yeah, for for for time series
38:52
forecasting, the one of the most common technique is called back
38:56
testing. I wonder if that's what you are
38:59
you were thinking. So basically you can pretend
39:04
this is who you were. You were sort of saying the past
39:07
at some time like and then at that time on you run a model
39:11
training and then you pretend this is OK what what the models
39:16
weight's going to be. And then you start to like
39:19
following the time like a simulation moving forward and
39:23
then generate forecasts and because actually everything has
39:27
already happened in real life. So you can actually see what the
39:31
model predicted versus what the history actually was observed.
39:37
And you can calculate the bias or any sort of metrics,
39:41
performance metrics for your accuracy.
39:44
So that's how we sort of compare models or every time we're
39:47
trying to make tweak into our models or evaluate new models,
39:53
back testing is what we do to sort of help us decide OK, if
39:57
this model actual perform better than another one.
40:00
How big of a factor is it that these models are also having a
40:06
geospatial aspect to them? Simple answer, it's very
40:12
difficult and so yeah. So for DN, right?
40:18
If it's single time series, you do long short term memory and
40:21
you don't have to worry about how data correlated spatially.
40:26
So model is simpler, less model weights, training faster,
40:31
forecast faster for spatial models.
40:35
Now, another thing people actually need to do is to sort
40:38
of dealing with the correlation between different spaces,
40:42
because you can treat every cell as its own model, but then if
40:47
there's 3000 cells, you have 3000 models.
40:50
You do not do that. So more practical solution is so
40:54
sort of treating the whole cells as, sorry, the whole region as a
41:01
sort of different dimension in a single input.
41:05
So the spatial temporal paper from Microsoft lab that I
41:09
mentioned earlier, what they do is actually they use some
41:11
convolution convolutional neural network approach.
41:15
So, so basically every time snapshot you have say what's
41:21
your demand is, what's your supply is and at different
41:24
locations and you can consider that as a image.
41:28
So your latitude is your like accesses in your image.
41:34
Your longitude is the Y axis in image.
41:36
Every cell is some sort of value on your image.
41:40
That's how image processes processing is basically handling
41:45
their data. So we saw we can use the same
41:47
approach. And in CNN, you, you can't apply
41:52
us like a set of kernels to sort of basic learning the spatial
41:56
correlation like a three by three cells, 15 by 5, sorry, 5
42:02
by 5 cells to get their correlation across the space and
42:09
applying different weights. So we can learn like, OK, maybe
42:12
when this guy is going up, that guy's going up like, oh, when
42:17
this location have a big demand, the other location have a big,
42:21
the nearby location have a big demand or opposite.
42:24
So it's learning all these spatial correlations.
42:26
So you end up having model weights in your neural network
42:30
to capture those informations. But to learn those weights well,
42:36
which means you need more training data set.
42:40
And although one thing we learned is those correlation can
42:45
change over time. So you're trying to get learn
42:49
more about the correlation, but then if last year's correlation
42:52
is different from this year's correlation, it does not do
42:56
well. You sort of can only learn a
42:58
limited amount of correlation. And this could be like some kind
43:03
of construction that's happening for half of the year.
43:07
And so you're creating a correlation that is only because
43:12
of the construction, right? Yeah.
43:15
And also maybe you know, some offices people relocate to
43:20
another building or pandemic, that's a huge hit, right?
43:23
People all of a sudden travel differently and people moving
43:26
constantly from location to location.
43:29
Economy, macro economy can affect like a strip mall sort of
43:35
running out of business and changing people out of control,
43:38
although let's say those are less issue.
43:42
But yeah, you just you can only use a limited amount data in in
43:47
that sense. So I could see how the creating
43:50
patterns for the spatial data. For example, in my head, I play
43:58
it out as if there is a stadium and you have an event happening
44:03
at that stadium, it's the north and the South side that people
44:06
are going to be entering. So you would expect that there's
44:09
some correlation there that when one side is busy, there's also
44:13
the other side busy. Yeah, that that can happen
44:17
sometime. That actually remind me a very
44:20
interesting learning. We also learned is venues they
44:24
operate differently time from time somebody jumping OK, last
44:29
month we did this with these two like entrance right SN so that's
44:35
how they're you're directing people to depart from venues.
44:40
That was like OK, that was horrible.
44:41
We that was a mistake. Let's change like let's make ES
44:46
plus some other location and one month they're like let's do a
44:50
bus shuttle like so to move people from this location to
44:53
another next month. You know shuttle service is not
44:56
reliable. People still complaining so the
44:59
venues interesting keep changing too that also have unlocked at
45:03
airports like like sometimes they use lot A for like Uber,
45:09
Lyft pick up. Sometimes they use lot B and
45:13
that changing over time too. So that keep messing up our data
45:18
unless we know exactly how the venues or the airports being
45:22
operating. But sometimes it's hard because
45:24
there's always an information delay.
45:26
Which also makes sense if you think about a football game
45:31
versus a Taylor Swift concert and how the crowds of these
45:39
events operate and then the times of these events.
45:42
And so maybe you're thinking, OK, it generally when there's an
45:47
event, if you just have it as event at stadium and you don't
45:53
know more information about that, is it a football game or
45:55
is it a concert, then you're going to get burnt.
45:58
And so you probably have to have more granularity on to what type
46:02
of event is it. Yeah.
46:04
And yeah, that, that's definitely so true.
46:08
And sometimes it's hard. And also like the same singer
46:12
sometimes might attract different crowds.
46:15
Sorry, the same by different singers, they might try attract
46:18
different crowds and certain crowds might more leaning
46:21
towards riding an Uber, Lyft, certain crowds might prefer or
46:26
like driving themselves. So we also noticed that too,
46:29
like sometime it's the same venue, same time the week, but
46:35
somehow they have very different demands.
46:39
That's where your forecast model, I'm guessing it just gets
46:42
blown out of the water. And so that's what you're
46:44
talking about where you need to kind of look at the the misses
46:50
or the negatives instead of looking at what it is doing
46:54
correctly. Yeah, that's like the refitting
46:57
or bias adjustment is so important because we can't, we
47:01
know we can't model them correctly.
47:03
And so the best we can do is learning from our mistakes.
47:06
OK, last five minutes beyond the forecast, but this much.
47:09
So let's do a refit of model or let's apply some bias correction
47:14
because sometime we look at the errors that we don't even know
47:17
why we see this big bias in our forecast.
47:21
It's it's just mind blowing. Yeah.
47:23
So, yeah, just like in reality, it's like so by simpler model
47:29
interesting, make it a little bit easier because you can't
47:33
apply adjustments easier and faster versus complicated model
47:36
DNS. It's it's erratically very
47:41
interesting, but if you're dealing with especially when it
47:46
comes to datas related to human behavior that tend to come
47:52
eluded with many other unknown factors.
47:54
So it's very hard to really forecast them well versus I
47:59
would say like if you're dealing with image processing or
48:02
language, those are more structured, right?
48:05
People talk following certain grammar, you have some
48:08
variation, but then it does not really fall too much outside the
48:13
box. So that's why it's more
48:14
predictable because it's not so outside the box.
48:17
So it's probably easier to train them well.
48:20
And also you have so much data like like language.
48:25
Oh, I think when the language change like like totally will
48:29
probably take thousands a year, right?
48:31
You can also start talking differently, using different
48:34
idea concept. At least for the last 5-10
48:38
years. I wouldn't say we will talk that
48:40
much differently, but when it comes to travel, sort of human
48:44
behavior is like, oops, five years ago, it's a totally
48:48
different word. You can't use that data to train
48:50
a model anymore. Yeah, and the you say that, but
48:57
you obviously have not spoken to many Gen.
49:00
Z people, I guess if you don't. That's true, yeah.
49:05
The no. But if you compare Shakespearean
49:08
language too, our language, yeah, that is a monumental
49:12
shift. But that also took hundreds of
49:14
years or 1000 years, I can't remember how long ago that was.
49:19
And so if you're looking at this there, there was something that
49:23
I wanted to ask you, though, about how often you're trying to
49:27
root 'cause these anomalies that will come through.
49:31
Like you said, sometimes you just don't know.
49:33
Why was there so much bias in this prediction?
49:37
Why did we not get it right? How much of your day goes into
49:42
figuring out why you didn't get it right versus just business as
49:48
usual? Yeah.
49:49
So it depends on the use cases. So I think my team, since we are
49:57
looking at cell level like millions of data points, we
50:01
don't really have that much capacity to really look into
50:06
that level. So what we typically do is we
50:08
have certain metrics at more like regional level, like
50:12
aggregate all cells together, look at the forecasts versus
50:16
actual. We have certain metrics we trend
50:20
if it's like the values is too far off or if there's some sort
50:23
of drifting the trend, a sets of sort of metrics we monitor if
50:29
they trigger certain alarm, that's the time we sort we'll
50:33
spend effort to look into that. Other time is typically because
50:37
all of these actually goes, goes into downstream models like
50:41
pricing, right or dry incentives.
50:43
So they have their performance metrics on their side too.
50:49
Typically if they're meeting their sort of certain pricing
50:53
targets, if their price is somehow their model just start
50:56
to spitting out like price that's way outside their bounds
51:02
or something, they will look into that and see if that's
51:05
triggering by forecast or other factors.
51:07
If they do believe it's something on the forecasting
51:10
side and they will come back to us and we also look into that.
51:15
So that's actually how things are sort of set up for the real
51:19
time site due to the granularity, I think Challenge,
51:24
yeah, but for offline forecasting, that's not my team.
51:30
But there's other cases where they do just regional level
51:34
hourly value for an ex say to one week or two, they they have
51:39
people sitting down review like region by region.
51:43
So that's actually a more like frequent cadence on on their
51:49
side. Yeah.
51:50
And but then they typically are very sensitive to the
51:57
differences because that's actually a team.
52:00
They're downstream typically sort of decides the company to
52:04
decide, OK, how much money they want to spend next week.
52:06
Because if there's a big gap in incentives or sorry, if there's
52:11
a big gap between supply and demand, they need to decide, OK,
52:15
how much money they need to spend to acquire drivers or
52:17
acquire riders. And those are like a big chunk
52:20
of money. Yeah, there's big decisions
52:24
being made on that. So you want to make sure those
52:25
decisions are correct. Yeah.
52:27
And also I think it's technically it's also more
52:30
feasible. Typically you can't look at top
52:32
regions or you can't look at regions that you know there's
52:36
going to be big events going on. So we can easily check, OK, if
52:38
we're forecast during that time at that location is accurate.
52:43
I guess we can do the same thing as cell level if we want to say,
52:47
OK, this is the event. But then our forecast actually
52:51
happens every minute for next 30 minutes or an hour ish.
52:57
So the only time we will see that forecast is like 30 minutes
53:00
before that. And so I don't know if that's
53:03
feasible If the events happens at midnight, I don't know if
53:07
we're going to wait till like midnight and quickly evaluate
53:11
the performance. So everything is automated.
53:14
We are more looking at OK as the forecast happening that we
53:18
observe the actual if the difference is bad or not.
53:21
If it's bad, then let's do adjustments in the model like
53:24
ASAP. So that's a sort of very
53:26
different concept.