Latest / Agentic Conversations (formally mlops.community) / AI in Education Fireside Chat // LLMs in Production Conference 3

Transcript

AI in Education Fireside Chat // LLMs in Production Conference 3 · Agentic Conversations (formally mlops.community)

0:04
Maybe before I I, I kick off the panel just to say a couple of
0:08
things about why we thought we we want to spend a little bit of
0:12
time with with you all on AI and education at process.
0:16
As you might have heard us say now a couple of times, we invest
0:20
in different sectors. One of the big ones is the
0:23
Edtech, basically education technology companies, there's
0:26
about 12 of them. And this is a space that we've
0:30
been very excited about for some time because we believe here
0:33
technology can play an important role into improving the way
0:36
people learn all around the world.
0:38
And so in that work we had had the privilege of working with
0:42
leaders in in in some of the companies and and others that
0:46
are trying to basically change how we learn using technology.
0:49
And we thought to invite three people to basically have that
0:54
discussion with us today. And I want to quickly maybe just
0:58
introduce them or at least say who they are and then I'll let
1:01
them introduce them and and say how they're using that in their
1:03
company. So first we have Eva Houston,
1:05
who is the founder and CEO of Solo.
1:08
Learn welcome Eva. And we also have Bill Salak, the
1:13
CEO of Brainly and Clinton from Duolingo heads up the AI work at
1:19
the Duolingo. And I think between the three of
1:22
you, we're close to representing about a billion learners that
1:25
you each serve through your company.
1:28
So that's quite an interesting, I think number and each of you
1:33
has thought about how you bring AI into your products in
1:36
different ways. So I'd love maybe for you to
1:39
maybe quickly introduce yourself what you and the company have
1:44
been looking at and maybe what you're most excited about in
1:46
terms of the one or two AI use cases you've been spending time
1:49
on recently. So maybe can I ask you to start?
1:55
Yes, sure. So thanks for having me.
1:59
Really exciting. So we're solo learning.
2:03
We are the most engaging platform to learn to code for
2:06
mobile first bite size, practise heavy with a strong community
2:11
that kind of supports you whenever you get stuck at hop up
2:14
on Google Play and App Store. Over 30 million registered users
2:18
around the globe, mostly organic growth.
2:21
So that's what Solar needs. And in the last year and a half
2:27
we have been experimenting with AI kind of you know just trying
2:31
to understand how AI improves learning experience in, in
2:35
general because as a company we're very much focused on
2:38
experience, you know how you teach learning science at the
2:42
core of everything we do and we have like we have productionized
2:48
you know an AI assistant, we have tested couple of things
2:51
that haven't worked and we discontinued them and a very
2:55
exciting project we're working on now and I think that's like
2:59
very much for the like the disruption in ethnic will
3:03
happen. Like if you look at what has
3:05
happened in ethic in the last 10 years, the biggest focus was on
3:10
has been on content generation. So we have generated you know
3:13
tonnes of video lessons on any topic out there where attack
3:17
hasn't been super efficient like other than like a couple of
3:21
companies you know, just investing in the space is
3:25
basically you know, just replicating a great classroom
3:29
experience. Online.
3:31
And when you think about, you know just all the promise of AI
3:36
out there, which is, you know, an AI tutor for all of us.
3:39
You know, like personalised learning.
3:41
You know, like playground where you can test any skill you know
3:45
on the go. Or you know, AI books that can
3:48
teach us to be more, you know, socially and emotionally
3:51
intelligent. Those are all components of like
3:54
the great components of great teaching and great classrooms.
3:58
And I feel that's where it kind of, you know, just teaching AI
4:02
how to teach is a very like under, you know, like overlooked
4:07
area in a way and by just, you know, kind of, you know, just
4:11
investing in it. And that's a project I'll talk
4:13
more about, but that's like a very exciting project.
4:15
We've recently started basically training the models not only to
4:21
produce content but also to produce it in a format that
4:24
actually teaches. So super excited about that.
4:28
It's not live yet, but we're seeing very good early results
4:31
and I'm going to stop here. So fantastic.
4:33
Yeah. We're definitely going to spend
4:35
a little bit of time going a little bit deeper on that in a
4:37
second that you have a thanks for for the introduction and
4:39
thanks for joining us today. Maybe I can quickly hand it over
4:43
to you, Bill, and say a little bit about, you know, what you do
4:46
at Brain Lee, what the product is and what are some of the use
4:49
cases that you've been working on recently?
4:53
Absolutely. So Brandley is a community
4:57
learning platform. You can think of us as the
4:59
world's largest study group. We have hundreds of millions of
5:02
learners that come to us every month to get help.
5:05
We have an extensive, extensive knowledge base that covers all
5:08
school subjects and grades. All of this is very
5:12
sophisticated technology. It's a platform that helps
5:15
learners get unstuck when they need help.
5:17
We integrate AI into that experience.
5:19
We also integrate community, so it's learners helping learners.
5:23
AI is there to assist when needed and our goal is to give
5:28
learners high quality explanations to the questions
5:32
that they're struggling with. So this is schoolwork, but we're
5:35
helping learners outside of school.
5:39
Our, our goal or the thing that's driving us right now and
5:42
where we see integration in in AI and sort of where our biggest
5:46
technology investments are going right now is all around this
5:49
concept of giving the explanations that your teacher
5:53
would have given. So we we don't just think about
5:56
giving an explanation or giving an answer, but we think about
6:00
what are the qualities of an explanation or an answer that is
6:03
most suitable for this particular learner.
6:05
Like what is the vocabulary being used in their classroom?
6:09
What are the concepts that they've been exposed to, so that
6:11
when we give them an explanation, it actually serves
6:15
their needs. It's not that we're introducing
6:17
new words they've never heard before or concepts that they
6:20
haven't learned yet, But we're really trying to get as close to
6:22
the classroom as we can to provide that, that level of
6:26
personalization that you get from your teacher who's giving
6:29
you a one-on-one, you know, sort of tutoring session, so to
6:33
speak. There's a lot going on in our
6:36
platform to be able to make this happen, but but all of the
6:39
features, all the things that we work on are really around this
6:42
particular goal. Yeah, thanks for sharing that.
6:45
And you know you and the team were, you know, some of the
6:49
earliest adopters you know, in the group and sort of trying to
6:53
embrace how AI could I think help you in that North Star.
6:55
And I think we'll talk about this hopefully in a bit as well.
6:58
But you've certainly helped us understand what the the the high
7:03
quality aspirations are that you also put yourself and the AI and
7:07
the community too when it comes to providing the right answer.
7:10
And when you know how you think about evaluating answers,
7:12
whether they come from your subject matter, experts or AI.
7:15
And it isn't just the correct answer, but it needs to
7:17
basically be as good as you described, as the teacher needs
7:20
to be. So we'll go deeper into that in
7:23
a second, Clinton. So thank you for joining us from
7:28
Duolingo. Maybe the same question to you
7:30
and also maybe you can share a little bit.
7:34
Well, you know how you spend your days in Duolingo.
7:35
What are the key use cases that you're, you and the team have
7:38
been, you know, spending time on when it comes to using AI and
7:41
and and giving a better user experience for the Duolingo
7:44
learners? Yeah, absolutely.
7:47
I'm excited to be here. Thanks for thanks for the
7:49
invitation. So, yes, I'm Clinton Bicknell,
7:52
Duolingo's head of AI Duolingo. You know, primarily we have a
7:56
language learning product to make sure everyone's aware of
7:59
that. We are also recently expanding
8:01
to some other subjects like math and music.
8:05
But yeah, primarily most of our AI is happening in the language
8:07
learning space. We've we've been we've been
8:10
using AI since close to the beginning of Duolingo, like a
8:12
decade ago, largely for personalization related issues,
8:17
right? Trying to figure out what are
8:18
the right exercises for this given learner at this given
8:20
time. I think maybe most relevant to
8:25
today, we, you know, most recently some of the cool new
8:29
things that we've started doing with large language models.
8:32
The latest generation of them include one big area is a lot of
8:38
interactive kinds of features, right?
8:40
So being able to like have a conversation in a language that
8:43
you're learning is something that the technology wasn't there
8:45
to support before. Now we we are building various
8:49
features where you can have different types of conversations
8:52
there. There's a lot of interesting, a
8:53
lot of interesting problems that come up there in terms of how to
8:57
make that actually a compelling feature for learners who are
9:00
wanting a sort of a fun bite sized experience who are maybe a
9:05
bit less like self-directed. We are also doing some work in
9:10
the explanation space. So interesting to help hear what
9:13
Bill was talking about. There definitely, definitely
9:15
some connections because, you know, now we can actually give
9:18
people explanations of if they get something wrong, you know,
9:21
why, Why was that wrong? They are, you know, mostly
9:25
right. But there's a lot of
9:28
optimization we're still doing there.
9:31
And then we are also, you know, using a large language models
9:34
quite a bit echoing back to something that Givea was talking
9:38
about in terms of thinking about how to automatically, you know,
9:42
generate parts of the curriculum or figure out, you know, how to
9:44
teach a particular concept to a given learner.
9:47
And and I certainly would echo what what she was saying there,
9:50
that, you know, generating content as content is easy.
9:54
Generating content that teaches well is a very different problem
9:58
that large language models don't know how to do out-of-the-box
10:02
except in kind of obvious common sense ways.
10:04
So yeah, a lot of a lot of other interesting AI work happening,
10:08
but I think maybe I'll focus on those for today.
10:11
Great. Well, thanks.
10:12
Thanks for sharing that. And I remember, you know, in my
10:16
days when I was doing my PhD at Carnegie Mellon, I think that's
10:18
that was when Duolingo was born. And at that time there's, you
10:22
know, some of the early ML work was taking place there.
10:24
So it's not really surprising to hear you have that in your DNA,
10:28
it sounds like now you might have seen that the theme of the
10:32
of the event today is LLMS in production.
10:35
And you know there's a lot of builders out here joining today
10:39
listening to to the panel and various speakers have shared
10:43
very sort of you know, things that have brought a little bit
10:45
of realism back into the discussion.
10:47
You know, agents suck for example or how hard it is to do
10:51
evals and so on. And for all of the promise and
10:55
so on that we of course think about when it comes to AI and
10:59
and being applied in education. I'd love to also ask you, you
11:02
know, what are the, what are some of the things that you've
11:04
learned, lessons learned maybe where you had expectations on AI
11:09
moving, some important, you know, metrics that you've been
11:12
looking at when it comes to learners and and the way they
11:14
use your product that weren't exactly met or other sort of, I
11:18
guess you know, because many of you are early movers in this
11:21
space other you know, wisdom you've you've gained in the last
11:25
18 months as you sort of thought about or maybe even before but
11:28
thought about Brit building this next generation of MO models
11:31
into learning. Maybe, Bill, you have some
11:36
thoughts on that one? Yeah, absolutely.
11:39
So I mean the audience should know Paul that you and I, you
11:43
pulled me into this pretty early like you I think you were the
11:45
one who brought open AIS new GPT models to me and said hey you
11:49
got to look at this and I I can say until you showed me what you
11:53
guys are working on, you know we had, we had dabbled in the space
11:56
and I was quite sceptical. So.
11:58
So when we say when you ask the question like what disappointed
12:01
us or what didn't quite live up to our expectations, like we I
12:04
came from a place of very low expectations.
12:07
And I think I was you were consistently like pushing my
12:10
boundaries and showing me like, Oh no, this can actually work.
12:13
But I think what we learned, I think like the wisdom here that
12:17
that's most important to take away and certainly has worked,
12:20
has served us well. If you're relying on the LLM as
12:25
a fact generation engine or you're looking at it as a body
12:28
of knowledge, like that's where you're going to find
12:30
disappointment. Where we find the biggest
12:32
successes is in using to synthesise or produce a an
12:40
explanation or text or an output that is informed by something
12:44
we're able to give it. So we put it on various rails,
12:47
right? We put it on On.
12:49
We put it down a path by using augmented prompts or we feed it
12:54
information, basically ask it to synthesise this information into
12:58
a more productive or human like response.
13:01
And I think it does really well at that, where you give it space
13:05
to go off the rails, where you give it space to kind of figure
13:08
things out. I think that's where you're
13:10
going to find disappointment. It's also the least competitive.
13:13
So think about your data. Think about what you know about
13:16
the problem that you're trying to solve as your competitive
13:20
advantage in the market space. Anytime you rely upon the LLM to
13:24
sort of do what everybody else can do with it without
13:27
augmenting it with your special sauce, I think you're you're
13:30
sort of competing at the lowest common denominator and you're
13:33
not going to have much success with that.
13:35
That can serve us in a number of ways, right?
13:37
Not just facts, but just in producing responses that are
13:41
sort of generic and not necessarily helpful in a
13:44
personalised use case. Thanks for sharing that.
13:47
Maybe a follow-up question just to before we go to to Clinton
13:53
and give up. How do you address this right,
13:55
because you talked about hundreds of millions of
13:57
learners. You do have this in your
13:58
product, You do have, you know these basically it's called
14:02
false positives or mistakes, misleading answers and so on.
14:04
That wouldn't live up to your standard.
14:06
How do you deal with that brainly?
14:10
So you know with a with a healthy dose of scepticism that
14:13
we entered this in, we really started by putting the features
14:17
that we created in in serious constraints.
14:20
So we didn't ask it to generate responses to questions or
14:24
explanations to questions. We started with something simple
14:26
like here we have an answer to a question but it's not serving
14:30
our learner and the learner will give the learner the ability to
14:34
have this explanation reframed. So we added a simple feature
14:39
like simplify this explanation or explain this in more detail,
14:43
right? So now we have the context, we
14:44
have the facts. We're comfortable with that, but
14:47
we're able to modify the use case.
14:49
The user is able to modify the use case to serve their need.
14:53
Only later did we get into retrieval, augmented generation,
14:57
use cases only after we felt comfortable with sort of those
15:00
simpler things that were more constrained, but the rack stuff
15:04
really expanded a new area. And then when getting directly
15:08
to your question, you know, the more you know about your
15:11
learners, the more you know about their behaviour patterns
15:14
and their curriculum and their content.
15:16
And it should be. I should be clear, we don't
15:19
teach the we don't own the curriculum at Brandon.
15:21
We are not the ones who have generated the curriculum.
15:24
We are literally a tool for any school system, any curriculum.
15:28
So when when learners are asking us questions, we are trying to
15:31
learn what they're learning. We're trying to figure out which
15:33
cohort of learners looks like them, what's the content they're
15:38
most likely to visit, what are the patterns in our knowledge
15:40
base that that we can apply to them to then be able to augment
15:46
the LLM generated output. So it is about understanding
15:50
your audience. It's it's about understanding
15:52
their behavioural patterns and the content that they're going
15:55
to be traversing now and in the future and then augmenting those
15:59
prompts. That's a very, very difficult
16:02
thing to do. But I like to simplify it for my
16:05
stakeholders by saying, Oh yeah, I know this is the same thing
16:07
you see on Netflix and YouTube. This is just, you know,
16:11
commonplace stuff. It's it's actually quite
16:13
difficult. I would definitely agree.
16:16
It's brave to simplify that the risk of then being told to just
16:19
do it fast and everywhere, but that's thanks for sharing that
16:24
Eva, maybe the same question to you.
16:25
So, you know, I think we work together for some, some time and
16:30
I think we're excited. Into the space too, so.
16:33
Yeah. So but at the same time we
16:35
learned some things along the way.
16:37
But I'm also curious how you see that now as you're shaping the
16:41
direction of the company or if you look back the last 18
16:43
months. What are some of those lessons
16:44
where you said these are expectations I had that maybe
16:47
didn't sort of turn out to be exactly as matched and and and
16:51
how you use that information if you decide what you do next?
16:55
Yeah, I like to echo whatever Bill said because I think like
16:59
we started with, you know, let's just do something, you know,
17:03
everyone is talking about AI, let's, let's get into the space.
17:08
And what we have learned is like exactly the same, you know
17:11
because you're not going to be differentiated if you build just
17:15
another assistant, you know it will be somewhat helpful
17:18
probably it will solve some problems, but it's not going to
17:21
be you know a core like basically a core part of your
17:25
product. And what we, we we think we do
17:29
best and like we have a lot of room for improvement is we teach
17:33
and we started the company you know with like can you make hard
17:37
things easy to understand. So that was the big premise and
17:40
the content work we're doing right now is exactly focused on
17:44
that. And the exciting part of it is
17:47
that we have actual teachers who kind of, you know, just provide
17:53
feedback and we train the system based on the teacher feedback
17:56
because we have learned that that's the part that is missing.
17:59
Because yeah, AI is great, you know, to produce like a lot of
18:04
information, you know, just just like the factual information is
18:08
Bill put it. But then it's not great in kind
18:10
of putting it in a way that that students actually understand.
18:13
So we started with very small pieces of our bite sized
18:16
content, you know, just the theory part first.
18:19
Then we moved to like simple practise exercises and the
18:23
deeper we go like like so like basically I think that's where
18:29
like we can make some difference in in automating what we know
18:33
about how people learn. Cool.
18:36
Yeah. I think that some of those
18:38
lessons were hard learned. I remember also when we were
18:41
actually excited about some of the feedback rid of the small
18:43
cohort, we did the math on the cost and we said, wait a second,
18:46
we can't really scale this up. And so I discussed with the team
18:49
how we could train our own model and eventually we did to kind of
18:53
make sure it was also cost effective.
18:54
I think that comes back to one of the themes we started with,
18:57
you know, at our scale, at you know your skill, this becomes a
19:01
serious consideration. So.
19:04
But that feature that we built together is is is a is kind of
19:07
loved one. Like, it's not a core feature,
19:10
but it's it's very much used, so it explains the coding errors in
19:14
the human language, so it makes it easier for students to
19:18
understand the introduction. And it is kind of, yeah.
19:21
Yeah, it definitely gets to the point of making complex things
19:23
easier to to learn. Easy to understand?
19:25
Totally, yeah. So Clinton, you know, given that
19:30
Duolingo is also, you know, in in in your DNA, how how do you,
19:34
how do you look back at some of the lessons you learned in, you
19:37
know, jumping into some of these ML applications and and things
19:41
that maybe didn't turn out quite as you expected either on the
19:44
metrics or on other ways to kind of bring this into users that's
19:47
at your skill? Yeah, Yeah, great.
19:51
Great question. So yeah, I think I would, I
19:53
would echo some of the stuff 1st that the building that we're
19:56
talking about. One thing though maybe turning
20:00
this into slightly more of a discussion I I took some of what
20:05
Bill and Neva was saying to kind of be opposites actually if I
20:08
under if I was following correctly.
20:09
I think Bill was one of the points he was making was that
20:12
generating the getting AI models to generate the underlying facts
20:17
can be very hard to get them to do it right.
20:19
Instead like you generate those based on your own internal data.
20:24
And then use AI to kind of package that in a nice you know
20:27
way to to present to learners. And if I and I think, yeah, that
20:31
was actually kind of saying the opposite right, that use the AI
20:34
models to generate the facts and then and then the hard part is
20:38
actually figuring out how to like present this in the most
20:40
useful way to a learner that I that I follow those things
20:44
correctly. Yeah, but I didn't mean that
20:49
generating facts is always going to be correct and great and you
20:52
know, so you have some work to do, I think on both ends.
20:56
Yeah, No. OK.
20:57
That that's interesting. Yeah.
20:58
So, definitely we've also found that, yeah, the fact part is the
21:01
is the hardest part to get to get right most of the time.
21:05
I mean sorry, it's hard to make it such that that's almost
21:08
always right. We definitely run into some run
21:12
into some hurdles there. Several of the solutions Bill
21:14
mentioned are things we've explored.
21:16
I was also going to mention in terms of what another thing that
21:20
we've realised in this modern LLM world doesn't work as well
21:24
is, is just naively running these things at scale because of
21:27
cost. And I guess Paul you just
21:29
brought that up right before. But but yeah, that that seems
21:33
that seems very true at at you know, very large scale that that
21:37
we operate at even you know even some of the cheaper models are
21:42
are still you know cost prohibitive to run, you know to
21:45
run on you know every user response or something like that.
21:48
We've actually been doing a lot of work in figuring out ways of
21:52
using large language models in in ways that don't scale with
21:58
with usage. So for example, you know using
22:01
them to pre generate a lot of possible options for example.
22:04
But then you're just picking from in certain cases or or even
22:08
in some interesting cases we are doing things like using using
22:15
large language models to generate cheap like rules that
22:19
we can apply in real time to like detect certain things like
22:24
if a person made a particular kind of grammatical error or
22:26
something like that. And so that that's actually been
22:30
very interesting to where we're finding ways of using these
22:32
models to generate cheap detectors that we then run
22:35
without those models across across everyone.
22:39
That's a very cool one. I think that's something we we
22:42
end up talking about as well. A lot.
22:44
Because you know generations to do good generations is
22:48
expensive, right? You need these really high
22:51
quality large models but you know, selecting between the best
22:56
answer basically ranking is a much lower effort task right.
23:00
So you can either train your own models or find other ways to
23:04
have these models generate the rules.
23:06
So that's nice to hear. Hear that you guys are working
23:09
on that now we've you know I I purposely sort of wanted to dive
23:13
a little bit on the on the on the learnings and the scepticism
23:15
that you might have developed over time on where this might
23:18
and might not work. At the same time I think there's
23:20
also this is one of the areas where the potential for
23:23
transforming the way we learn is maybe more evident than in other
23:27
industries. So I would like to you know
23:30
maybe draw on your on your ideas for if you were to sort of plot
23:34
the next couple of years what are the things that you are
23:37
excited about right. So having tried and and and and
23:41
learned and and and and maybe stumbled on on some use cases
23:45
like what are the things that now given all those things you
23:48
feel will really shape the industry, you know you are
23:52
building together. So I don't know who wants to go
23:54
first. I see Clinton nodding.
23:57
So maybe you want to start? Yeah, yeah.
24:01
Big question, SO. It is.
24:03
It is a big question and I think there's a lot of different
24:06
exciting possibilities. I mean definitely one is just
24:10
for for language in particular is just enabling people to
24:14
practise and learn in a much much more interactive way, in a
24:19
much more kind of real time adaptive way.
24:22
So you know, being able to, you know, practise ordering a coffee
24:24
at a restaurant or something right is like a skill that you
24:28
can now practise a lot of times where you you just couldn't
24:30
before. So there's a lot of kind of step
24:32
change functions like that. I I think that 11 problem you
24:37
run into in these cases though, which which I think the
24:40
community is kind of starting to grapple with now, is that it's
24:43
pretty easy to just like make a chat bot.
24:47
But making an actual, like compelling interactive product
24:50
that is based on chat is a very different goal.
24:54
You know, something that people want to come back to every day
24:56
and keep keep doing things with is that?
24:59
An AI problem in your view or is that something that you say
25:01
we've got the AI tools for, but it requires rethinking the
25:05
product experience and the the way we've we're presenting these
25:09
models into our current journey? Yeah, it's a really good
25:13
question. I mean I think definitely
25:14
there's a lot of product work to do there for sure.
25:17
I'm trying to figure out like what what should these
25:19
experiences look like, you know what what do we want to build.
25:22
But I think then once once you nail that down for for a given
25:26
use case, there are still a lot of interesting AI problems that
25:30
that come up there. So for example you know one
25:32
thing that we're running into Duolingo, now we we have you
25:36
know one of our the main like interactive chat feature that
25:39
we're making is about it is basically like you know a goal
25:46
directed in chat like you know ordering a drink at A at a cafe
25:49
or you know or helping helping a tourist find their passport.
25:53
You know, something like particular like goal directed
25:56
conversations. What Even once we have that and
25:59
we want to generate things, we have several different
26:02
constraints and we're trying to generate these conversations to
26:05
be like with our Duolingo world. Characters have their own
26:07
personalities, obviously trying to, you know focus on particular
26:11
kinds of of topics that we're teaching.
26:13
But also one of the really hard constraints is just keeping the
26:17
language like at a level that a learner of this ability can
26:20
understand, right. And that that turns out to be
26:24
like one of the hardest problems.
26:25
And you can, you can very, you can do a rough pass and say
26:28
like, you know, ask. Basically, like in your prompt
26:31
you can say something like, oh, try to have this at the level of
26:34
a, you know, at the level. Of 1/3 grader.
26:36
Or something, whatever you say there.
26:38
But that only gets you so far, and there's a lot of other
26:42
interesting AI work to get it to actually be at the right level.
26:46
Yeah, that's a great point. So I think that's it.
26:48
You know, you know, I think in in in the language space we talk
26:51
about C1C2 of these levels. That's definitely something you
26:55
can tell a language model to do or adhere to very well today,
26:58
but maybe with fine tuning and so on, that becomes a easier in
27:03
the next couple of right. That's right.
27:05
Fine tuning is pretty helpful there.
27:08
But yeah, so I think one thing that's going to happen is just
27:10
figuring out like what? What kinds of really interactive
27:13
learning experiences resonate with people and are things that
27:16
people are going to be coming back to?
27:17
And how do you build those versus just?
27:21
Here is an open-ended chat bot that in theory can teach you a
27:23
lot of things, although it may make stuff up sometimes.
27:27
Yeah, that's a that that's a definitely exciting language
27:30
learning future. So hopefully we get to see that.
27:33
Bill, how do you think about based on everything you've seen
27:36
so far and and and the the, the, the, the feedback you're getting
27:40
from your learners, what, what excites you in terms of the next
27:45
coming years? You know my statement is not so
27:52
much driven by what I see happening with new products or
27:56
or new technologies. It's more about the acceptance
28:00
of these technologies in new places in our lives that we we
28:03
haven't been readily accepting them.
28:06
So when we, you know, when we first started working with LL,
28:09
Miss one of the literally like every interview that I did since
28:15
opening, I released Chat DVT. The question was and how do
28:20
people feel about having AI in the classroom?
28:23
Or how do people feel about AI interacting with their children?
28:27
As if you know, this was some version of the Terminator that
28:31
was coming for their kids. And so, you know, I think that
28:36
society in general, culture in general is is like more open and
28:40
accepting and starting to get comfortable with this.
28:43
So I'm really excited about the problems we can solve because
28:47
there are huge problems in education, informal education,
28:50
in the entrenched sort of systems of education that are
28:55
run by governments, national governments.
28:57
There are huge problems there. There's underfunding, there's
29:00
understaffing, there's lack of being able to have one-on-one
29:04
attention with kids, we can solve those problems with these
29:07
technologies. And that, to me, is the most
29:09
exciting opportunity in front of us.
29:13
Yeah, that that is hard to disagree with.
29:16
So I hope you guys can be part of that solution.
29:21
You have a same question for you.
29:27
Yeah, I think like there is a lot of exciting stuff like when
29:33
when you imagine where you can go, you know as a result of kind
29:37
of whatever was made available to us, you know the prospects
29:40
are like great and awesome. I think it will take some time
29:45
to get there. You know whatever we work on and
29:48
and we talked about it, it's not perfect and and it will need a
29:51
lot of tuning, fine tuning you know just a lot of kind of trial
29:55
and error and and learning on the go.
29:59
But yeah like we have no other choice because we need to be
30:02
learning on the go. We need to be upscaling you know
30:05
all the time just to stay competitive and to be relevant
30:09
and and so that's that's the part like the question is how
30:13
fast we get there and and yeah and help us to figure things
30:18
out. Yeah that's I mean hearing it
30:23
from from the three of you. I mean you've gone through I
30:25
think a healthy mind of scepticism to some excitement to
30:28
you know back and I still hear some optimism in what you're
30:32
describing in the future. So I guess as a as a father of
30:34
two young children I hope by the time they have to learn many
30:37
languages code or any other homework help, you guys have
30:41
built everything they need to do this better and more
30:43
effectively. So thank you for joining us
30:46
today. I really appreciate the time.
30:48
I know it's late for some of you early in other parts of the
30:50
world. So thanks for for joining and
30:53
sharing your your with.