Latest / What's Your Problem? / Can AI Tutors Help Kids Learn? Khan Academy Thinks So

Transcript

Can AI Tutors Help Kids Learn? Khan Academy Thinks So · What's Your Problem?

0:15
Speaker 1: Pushkin. There's a good chance you've heard of con Academy
0:23
Speaker 1: about how Salcon started out more than a decade ago
0:26
Speaker 1: tutoring his young cousins and then started posting simple tutoring
0:30
Speaker 1: videos on YouTube, like how to solve a quadratic equation,
0:34
Speaker 1: that kind of thing. He founded con Academy in two
0:37
Speaker 1: thousand and eight, and it grew into this big thing
0:40
Speaker 1: beloved by Bill Gates and the Ted talk set. Today,
0:44
Speaker 1: con Academy has tens of millions of monthly users and
0:47
Speaker 1: provides not just those YouTube videos, but thousands of practice
0:51
Speaker 1: questions across math and science and the humanities, along with
0:55
Speaker 1: software that monitors students progress. All this has been in
0:59
Speaker 1: service of a big idea using technology to give everyone
1:03
Speaker 1: the huge benefit that Salcon's cousins got from one on
1:06
Speaker 1: one tutoring. But in spite of Salza, the con Academy
1:10
Speaker 1: has not been able to fully deliver on that idea.
1:13
Speaker 1: The technology just hasn't been there to match what a
1:16
Speaker 1: human can do. But now sal thinks that may be
1:20
Speaker 1: about to change. Earlier this year, con Academy launched something
1:24
Speaker 1: they call Conmego, an AI tutor built on top of
1:28
Speaker 1: GPT four. I'm Jacob Goldstein, and this is what's your problem.
1:37
Speaker 1: My guest today is Sal Con. He's the founder and
1:40
Speaker 1: CEO of con Academy. Sal's problem is this, how do
1:44
Speaker 1: you use AI to bring students the benefit of working
1:47
Speaker 1: with a human tutor. Sal says his real work with
1:51
Speaker 1: AI started last summer when he heard from the co
1:54
Speaker 1: founders of the company that built.
1:55
Speaker 2: GPT About a year ago. Summer of twenty twenty two,
2:03
Speaker 2: I get an email from Sam Altman and Greg Brockman,
2:06
Speaker 2: the founders of open AI, who I knew, you know,
2:09
Speaker 2: I bumped into them at different events and things like that,
2:12
Speaker 2: and they said, hey, you know, we're working on our
2:14
Speaker 2: next model. If we would love to talk to you
2:17
Speaker 2: all about a few, you know, things that we might
2:19
Speaker 2: be able to collaborate on. I was skeptical that it
2:23
Speaker 2: would really have any implications for con Academy, but as
2:27
Speaker 2: a nerd, I said, Oh, this must be GPT four
2:30
Speaker 2: that they must be working on. And I have a
2:32
Speaker 2: lot of respect for Greg and Sam what they had
2:34
Speaker 2: already accomplished, so I was like, yeah, just I'm just curious. Yeah,
2:38
Speaker 2: let's meet is Actually at that point hadn't finished training
2:41
Speaker 2: what would eventually be GPT four, but they were about
2:44
Speaker 2: two weeks away from it, but they said, look, we
2:48
Speaker 2: think this is going to be the model, the generative
2:52
Speaker 2: AI that really opens people's minds to what's possible here.
2:56
Speaker 2: And because of that, it's going to be exciting and
2:58
Speaker 2: a little bit scary, and so we want to launch
3:01
Speaker 2: with some social positive use cases. The first one that
3:04
Speaker 2: came to our mind is education, and the first organization
3:07
Speaker 2: that came to our mind is Kin Academy, because y'all.
3:09
Speaker 1: Want they want you to be the warm, fuzzy, happy
3:12
Speaker 1: face of AI.
3:13
Speaker 2: Yeah, to see if we'd be interested to see and
3:16
Speaker 2: they want you know, they don't want to just be optics.
3:18
Speaker 2: I think they generally want AI to have warm, fuzzy applications.
3:22
Speaker 2: I believe, of course, the dark and sordid applications that.
3:26
Speaker 1: They clearly do, right, They clearly recognized the downside.
3:30
Speaker 2: Yeah, exactly. But the other reason they actually reached out
3:33
Speaker 2: to us and I connected the dots later, they said,
3:36
Speaker 2: you know, we really want it to be good at
3:37
Speaker 2: academic things like That's where GPT three it really had
3:42
Speaker 2: no solid handle on knowledge, and we think GPT four
3:46
Speaker 2: will and y'all have a lot of items we were
3:49
Speaker 2: just talking about how deep of an item bank we
3:51
Speaker 2: have it academy, across subjects and grades. And they were
3:54
Speaker 2: particularly interested in ap bio, which I later learned from
3:57
Speaker 2: Bill Gates. It was because when he saw GPT three
4:00
Speaker 2: he said he essentially told him, look, this is cute, guys,
4:03
Speaker 2: but I'll be impressed if this could pass an ap
4:05
Speaker 2: biology exam. So I think Greg and Sam literally said, Okay,
4:10
Speaker 2: you know, Bill Gates, that's a pretty good benchmark. Can
4:13
Speaker 2: this pass the ap bile exam? So I think that's
4:16
Speaker 2: the other thing. They said, hey, can we use some
4:17
Speaker 2: of your items to test to evaluate the model.
4:20
Speaker 1: So they wanted you have like whatever, thousands of questions
4:24
Speaker 1: about all these different subjects kind of test questions, test
4:27
Speaker 1: like questions about ap biology and calculus and lots of
4:30
Speaker 1: other subjects, and they wanted to train the model on
4:33
Speaker 1: your questions because you have the questions and the answers
4:35
Speaker 1: and exactly what you would want if you were training
4:39
Speaker 1: an AI model on sort of high school college level knowledge.
4:44
Speaker 2: Well, it's interesting out of the gate, they were actually
4:46
Speaker 2: more interested in just on the evaluation side. So once
4:49
Speaker 2: you produce a model, you want to see how good
4:51
Speaker 2: it is so you literally make it take questions and
4:53
Speaker 2: see how many gets right. So I was like, yeah,
4:56
Speaker 2: you know, maybe you know, I wasn't sure if this
5:00
Speaker 2: was really going to be something we could use, but
5:02
Speaker 2: I still wanted to see. I still wanted to see
5:04
Speaker 2: the demo when I was available. So two weeks later
5:06
Speaker 2: they said, oh, we finished training, we'd love to show
5:08
Speaker 2: you a demo. And I remember that meeting they put
5:11
Speaker 2: up and you know, this isn't that mind blowing the
5:14
Speaker 2: folks now because they've gotten used to this. But in
5:16
Speaker 2: summer of twenty twenty two, we were on a video
5:20
Speaker 2: conference and they on Greg's screen. He showed an ap
5:23
Speaker 2: bio question and he said, sah, what do you think
5:26
Speaker 2: the answer is? And I like read the question was
5:27
Speaker 2: about osmosis. I'm like, oh c. And then they asked
5:31
Speaker 2: the AI. I said yeah, see, I'm like, okay, that's
5:33
Speaker 2: kind of cool. And then I said, well, you know,
5:36
Speaker 2: ask it why is? The answer explained it very well,
5:39
Speaker 2: and that's when I was like, wow, that is interesting.
5:41
Speaker 2: I said, well, why aren't the other choices? The answer
5:44
Speaker 2: explained it very well. I'm like okay, yeah, this is getting.
5:47
Speaker 1: Kind of wow. Right, that's still like I know, it's
5:50
Speaker 1: not really cognition. Maybe it doesn't understand, but something is
5:53
Speaker 1: happening there, right, something something well.
5:56
Speaker 2: I think it's starting to make all of us start
5:59
Speaker 2: to parse words, like we start to realize that like
6:03
Speaker 2: sentience and cognition and intelligence are not necessarily the in fact,
6:08
Speaker 2: they aren't same thing, and we can separate these things.
6:11
Speaker 2: Something can be intelligent. And I think it's going to
6:13
Speaker 2: challenge our words, our semantics for some of these things.
6:16
Speaker 2: But then I said, hey, can it can write another
6:19
Speaker 2: question like this? And it did it, and I'm like
6:20
Speaker 2: that's a pretty good question. And then I said write
6:22
Speaker 2: ten more questions like this, and it did it. And
6:26
Speaker 2: you know, at the moment.
6:27
Speaker 1: At your moment where you were like, oh, no, AI
6:29
Speaker 1: is coming from my job, I kind.
6:31
Speaker 2: Of thought that a little bit. But then what really
6:34
Speaker 2: opened my mid They're like, oh, well, would you want
6:37
Speaker 2: access to it and to at least try it out
6:39
Speaker 2: see how you might be able to use it. And
6:42
Speaker 2: I was like, yeah, I want access. I want to
6:44
Speaker 2: try out this thing, and so myself and actually eventually
6:47
Speaker 2: our whole organization got under a non disclosure agreement with
6:49
Speaker 2: open ai back back in August, and we started testing
6:53
Speaker 2: it in that first weekend, and I remember it was
6:55
Speaker 2: myself in.
6:55
Speaker 1: The August twenty twenty two.
6:57
Speaker 2: Yes, myself, our Chief Learning Officer Christen, and our CTO
7:03
Speaker 2: Paul were the first people to have access to and
7:05
Speaker 2: I think we even got access to it before many
7:08
Speaker 2: folks that open Ai access to it, and we were
7:11
Speaker 2: just playing around with it, and we were also slacking
7:13
Speaker 2: with the open AI team. We're like, hey, you have
7:16
Speaker 2: any ideas. We're trying to get it to do this
7:17
Speaker 2: or we got it closed, but it's still and they
7:19
Speaker 2: were giving us some really good tips and by the
7:21
Speaker 2: end of that weekend we had it being able to
7:24
Speaker 2: take on personas and modeling you know, pretty good tutor behavior.
7:30
Speaker 2: And that's when it really when I started saying, Okay,
7:32
Speaker 2: this is a this is a game changer. This is
7:35
Speaker 2: science fiction has talked about artificially intelligent tutors forever, most
7:39
Speaker 2: famously Diamond Age Young Ladies. Illustrated primer Neil Stevenson wrote
7:43
Speaker 2: about this in the nineteen nineties about you know, an
7:46
Speaker 2: intelligent tutor being able to educate essentially. I mean, this
7:50
Speaker 2: book was set in the not too far future and
7:52
Speaker 2: it educates all these young girls in China living on
7:56
Speaker 2: barges were orphans and then they take over the world
7:58
Speaker 2: because this tutor was able to empower them so much so.
8:01
Speaker 1: Sidebar Neil Stevenson amazingly prescient, right like he's the like
8:06
Speaker 1: president guy for this, he's the president guy for for crypto,
8:10
Speaker 1: right like it's weird, right.
8:12
Speaker 2: Oh, he's pretty good. And he wrote, you know, diamond
8:14
Speaker 2: Age was nineteen ninety four, so yeah, you know, almost
8:17
Speaker 2: as an aside, it was a tablet app so he
8:19
Speaker 2: was prescient for tablet and mobile. But but that's when
8:23
Speaker 2: when I when I saw that it could take on
8:24
Speaker 2: personas and act as a tutor and not superficially but
8:29
Speaker 2: actually do some things that are I would view like
8:31
Speaker 2: actually quite thoughtful as a tutor, I said, Wow, we
8:34
Speaker 2: I think we're at the cusp of something here. And
8:36
Speaker 2: I didn't know if it was going to happen in
8:37
Speaker 2: like three months or three years, but I'm like, we've
8:39
Speaker 2: got we've got to work on this.
8:41
Speaker 1: Just to pause there and kind of go broader, Like
8:44
Speaker 1: the big idea behind con Academy the whole time, right
8:48
Speaker 1: since its inception whatever what more than a decade ago
8:51
Speaker 1: now is the power of a tutor right, that like
8:54
Speaker 1: a tutor is profoundly valuable, and that's sort of empirically clear, right.
9:00
Speaker 2: That's exactly right. It all got started back in two
9:03
Speaker 2: thousand and four with me tutoring cousins just on the side,
9:06
Speaker 2: one cousin Nada needed help, and then I started tutoring brothers.
9:09
Speaker 2: Wordspreads in my family, free tutoring is going on, and
9:12
Speaker 2: I saw it with my own cousins just on an
9:15
Speaker 2: anecdotal level, that everyone that I was tutoring, I was
9:17
Speaker 2: able to put in, you know, thirty forty minutes a
9:19
Speaker 2: day with all of them collectively, it was it was
9:22
Speaker 2: dramatically accelerating them. And so to a large degree, when
9:25
Speaker 2: I started making exercises and software for my cousins and
9:29
Speaker 2: then eventually making videos for my cousins that obviously many
9:32
Speaker 2: many more folks ended up using, I was always in
9:35
Speaker 2: the mindset of how can I help scale my tutoring,
9:38
Speaker 2: How can I start making that tutoring a little bit
9:40
Speaker 2: more self service so that my cousins and eventually other
9:43
Speaker 2: people could help themselves. So that's been the journey of
9:46
Speaker 2: coin Academy for the last I mean it's been almost
9:49
Speaker 2: it's been nineteen years. Since I started tutoring my cousins,
9:53
Speaker 2: and so this technology held the potential to take it
9:57
Speaker 2: that much further.
9:58
Speaker 1: Yeah, so you're playing with this thing. It's very early.
10:01
Speaker 1: The world doesn't even know about GPT for yet. What
10:04
Speaker 1: what are you doing? What are you working on? Within
10:05
Speaker 1: con Academy.
10:06
Speaker 2: At this point we started to see, Wow, this could
10:08
Speaker 2: just could really work, This could be really powerful, and
10:10
Speaker 2: we started figuring out ways to minimize some of the
10:14
Speaker 2: rough spots of Jenai, like hallucinations where it can make
10:18
Speaker 2: up things like it's making math errors, which is obviously
10:21
Speaker 2: a problem if we want it to be a math tutor.
10:24
Speaker 2: So we started going working through that. At the same time,
10:27
Speaker 2: we started having some pretty intense debates inside of our organization.
10:31
Speaker 2: Roughly speaking, half the organization was like, this is the
10:33
Speaker 2: most important technological advancement ever or at least in our lifetimes.
10:38
Speaker 2: We've got to go all in on this, like it's
10:40
Speaker 2: our duty. And then the other half of the team,
10:43
Speaker 2: not that they disagreed, but they said, look, kind academy
10:46
Speaker 2: stands for a lot it and we're there to help students,
10:49
Speaker 2: and if we go out there with something that's either
10:52
Speaker 2: not well baked or it leads to something that's suspicious
10:57
Speaker 2: or shady or just scary for folks. Because JENAI could
11:00
Speaker 2: be scary, it's going to be bad for us. So
11:02
Speaker 2: we started having those debates. But within a couple of months,
11:06
Speaker 2: as we were just kept prototyping, well, two things happened.
11:10
Speaker 2: One is I think a consensus. I don't know if
11:12
Speaker 2: it was a consensus, but I started to drive alignment
11:15
Speaker 2: around the idea like, these these fears and these risks
11:19
Speaker 2: that people are articulating are real. We should not ignore them.
11:22
Speaker 2: But there's ways to turn them into features that actually
11:26
Speaker 2: not only mitigate risks, but they actually can be enhancements. So,
11:30
Speaker 2: for example, we said, well, what if on what eventually
11:33
Speaker 2: would be konmego we didn't call it. Then, then all
11:37
Speaker 2: the sessions of a student who's under eighteen are recorded
11:40
Speaker 2: and accessible by parents or teachers. What if we have
11:43
Speaker 2: a second AI that moderates the conversations, and if the
11:45
Speaker 2: conversations go anywhere shady can actively notify the parent or teacher.
11:50
Speaker 2: What if our AI doesn't, It doesn't just give you
11:53
Speaker 2: the answer. Even before you know this was before chat
11:55
Speaker 2: GPT came out, we knew that this could be used
11:57
Speaker 2: for cheating that the raw technology could. We said, we're
12:00
Speaker 2: not going to do that, but it can, it can
12:02
Speaker 2: support you. So we started thinking through all of these
12:05
Speaker 2: what you know, we aren't going to make We aren't
12:08
Speaker 2: going to use information to train the AI, at least
12:11
Speaker 2: in this stage where people aren't sure how it might go.
12:14
Speaker 2: We weren't going to let any personally identifiable information go
12:17
Speaker 2: between the student and the artificially intelligent model. And so
12:20
Speaker 2: in a lot of ways, it's like we're gonna put
12:22
Speaker 2: more safeguards on jen AI than frankly there exist on
12:24
Speaker 2: the Internet, and when kids are just randomly on the Internet.
12:28
Speaker 1: That's a pretty low bar. You definitely want to be
12:30
Speaker 1: higher than random, let's be honest.
12:32
Speaker 2: Yeah, we made a much higher because we knew this
12:34
Speaker 2: was gonna kind of you know, people were going to
12:36
Speaker 2: have mixed feelings about it.
12:37
Speaker 1: A lot of ways it could go wrong. There's a
12:39
Speaker 1: lot of ways. That's right.
12:40
Speaker 2: That's right. Then, I would say the other big thing
12:42
Speaker 2: that happened end of November, chat GPT comes out, and
12:46
Speaker 2: that now that captured everyone's imagination and we all remember
12:51
Speaker 2: those first few weeks and months where people were you know,
12:54
Speaker 2: everyone was getting on chat GPT and taking screenshots and
12:58
Speaker 2: putting it on social media of what it was doing.
13:00
Speaker 2: And it was doing some amazing things. It was doing
13:02
Speaker 2: some very imperfect things too, the hallucinations, the math, theres,
13:05
Speaker 2: et cetera. I was worried initially because the narrative and
13:09
Speaker 2: education immediately became this thing as error prone, this thing
13:13
Speaker 2: can be used to cheat. It's the end of term papers, homework,
13:17
Speaker 2: et cetera. School district started banning chat GPT. I was like,
13:22
Speaker 2: oh no, we're working so much on this, and GPT
13:24
Speaker 2: four is so much better, and what was eventually going
13:26
Speaker 2: to become conmego was so much better, Like I hope
13:29
Speaker 2: the baby doesn't get thrown out with the bathwater. In hindsight,
13:33
Speaker 2: that was a good thing because it made us even
13:36
Speaker 2: internally say, look, the genie's out of the bottle. Now
13:39
Speaker 2: it's how we use the genie, Like we're going to
13:41
Speaker 2: be a force for good. Hopefully we have to work
13:43
Speaker 2: feverishly to show that it can be used well. And
13:46
Speaker 2: then I think by the time GPT four came out
13:48
Speaker 2: and Conmego came out with it in March, the education
13:52
Speaker 2: system and frankly society had a chance to process it
13:56
Speaker 2: and they had come around like, well, the technology isn't bad.
13:59
Speaker 2: We just need reasonable guardrails and we just need tools
14:01
Speaker 2: that are built for the actual use cases. And then
14:04
Speaker 2: we were able to show up with like you mean
14:06
Speaker 2: like this, you mean like con migo, and then most
14:08
Speaker 2: people and yes, exactly like that. And now we're seeing,
14:12
Speaker 2: I mean honestly, more schools and districts and parents want
14:15
Speaker 2: it for their students and their children than we can
14:18
Speaker 2: currently handle.
14:19
Speaker 1: So you mentioned some of the guardrails you built into it,
14:24
Speaker 1: you know, not accepting personal information, having a second AI
14:27
Speaker 1: monitoring for anything shady, not just giving answers. There's another
14:32
Speaker 1: piece of the process of building con Migo that I've
14:36
Speaker 1: heard you discuss elsewhere that is really interesting to me.
14:39
Speaker 1: And that's that's about having the AI sort of think
14:47
Speaker 1: about its answers, think about its responses. Right, I probably
14:52
Speaker 1: haven't articulated that well, but you can so tell me
14:54
Speaker 1: about that piece of it.
14:56
Speaker 2: As you can imagine, one of the hardest things to
15:00
Speaker 2: resolve was that even GPT four, which is dramatically better
15:03
Speaker 2: than GPT three point five at math, it was making
15:06
Speaker 2: an uncomfortable number of math errors, especially when it came
15:10
Speaker 2: to being in the tutoring use case where let's say
15:15
Speaker 2: there's an algebra problem and I the student take a
15:18
Speaker 2: step and maybe I distribute it, distribut use the distributive
15:21
Speaker 2: property and correctly will it recognize it? Will it not?
15:25
Speaker 2: How will it provide that feedback to the student? And
15:27
Speaker 2: out the box, it wasn't doing it that well. It
15:30
Speaker 2: was making mistakes a lot, unacceptably often. Then we had
15:36
Speaker 2: a it's.
15:36
Speaker 1: Really bad if the tutor is getting the problem wrong.
15:40
Speaker 2: No, tutor, Yeah, that's not acceptable. And then an open
15:45
Speaker 2: AI researcher gave us an idea, which is, instead of
15:49
Speaker 2: just having the AI respond immediately to the student, instead,
15:54
Speaker 2: what if you were to have the AI essentially on
15:57
Speaker 2: its own, not show this part to the student, but
16:00
Speaker 2: generate what it thinks could have been reasonable responses for
16:03
Speaker 2: the student HU and then use that plus the conversation
16:09
Speaker 2: with the student to then respond to the student.
16:13
Speaker 1: Yeah, and we did this thingly next level, right, like
16:15
Speaker 1: a teacher would always be doing that, but there's no
16:19
Speaker 1: reason to think that just a raw large language model
16:22
Speaker 1: would be right.
16:24
Speaker 2: That's right, But you know what's interesting about this. We
16:27
Speaker 2: did it and it dramatically improved the math. And then
16:30
Speaker 2: we started tweaking it more and more, and we got
16:31
Speaker 2: it better and better. Now we do a bunch of
16:33
Speaker 2: fancy stuff along along those lines to get it a
16:36
Speaker 2: lot better. But then we realize, to your point, that's
16:39
Speaker 2: exactly what a teacher would do. So that's what con
16:42
Speaker 2: Migo does. It works on it on its own and
16:44
Speaker 2: its own scratch paper, so to speak. Compares the student's
16:47
Speaker 2: response if they got something different. Con Migo doesn't assume
16:51
Speaker 2: the student is wrong. It says, because con Migo can
16:54
Speaker 2: sometimes be wrong even on its own work, It says, hey,
16:57
Speaker 2: I got something different, can you explain your reasoning? And
17:00
Speaker 2: then when the student explains the reasoning, that's really good
17:03
Speaker 2: for large language models. It's actually able to understand that.
17:06
Speaker 2: And what's interesting, not only is that very pedagogically good,
17:09
Speaker 2: we've been getting a strong a lot of feedback from
17:12
Speaker 2: students that they really appreciate that type of an interaction.
17:15
Speaker 2: It eerily feels human that this thing you were so
17:19
Speaker 2: used to computers being so perfect, and it's like you're wrong,
17:21
Speaker 2: you're right here, you know, that's what a robot would do,
17:24
Speaker 2: but no, this is what a tutor would do. It's like, hey,
17:26
Speaker 2: I didn't get the same thing. Let's work through it together.
17:27
Speaker 2: Let's see who made the mistake.
17:32
Speaker 1: In a minute, how conmego compares to GPT today and
17:36
Speaker 1: what it might be like a few years from now.
17:46
Speaker 1: When I was getting ready for this interview, I played
17:48
Speaker 1: around with con Migo and chat GPT. I put them
17:51
Speaker 1: side by side in different tabs in my browser, and
17:53
Speaker 1: I asked each one a basic calculus question, how do
17:57
Speaker 1: you take a derivative? This is a thing that I
17:59
Speaker 1: knew how to do a long time ago, but I
18:02
Speaker 1: forgot also a long time ago, so it seemed like
18:05
Speaker 1: a good test, and it was striking how different the
18:08
Speaker 1: response is were GVT gave me something like the Wikipedia
18:13
Speaker 1: entry about derivatives, a bunch of text with some rules,
18:16
Speaker 1: some equations, and then I tried it on conmego, and
18:20
Speaker 1: first it asked me what I knew, and I said,
18:22
Speaker 1: I knew algebra. And then it explained one rule, a
18:26
Speaker 1: sort of first rule, the power rule for finding derivatives,
18:29
Speaker 1: and it gave me a problem to try, and I
18:32
Speaker 1: got that problem wrong, and it sort of asked me
18:34
Speaker 1: what I was thinking, and I tried it again and
18:36
Speaker 1: I got it right. And then Konnigo asked if I
18:39
Speaker 1: wanted to try another problem, and I said yes, and
18:41
Speaker 1: it gave me another practice problem. So clearly this is
18:44
Speaker 1: a very different experience than the plain vanilla chat GPT.
18:49
Speaker 1: But it was still a little bit awkward and a
18:51
Speaker 1: little bit hard to follow. And when I closed the
18:53
Speaker 1: tab and came back, it didn't remember our previous conversation.
18:57
Speaker 1: So overall, very good, but not quite there yet. I
19:00
Speaker 1: asked sal if that seemed about right to him.
19:03
Speaker 2: Yeah, I think that's pretty accurate. I think where it
19:05
Speaker 2: is today, it's like if you really wanted to learn
19:09
Speaker 2: how to take a derivative, or if you're learning calculus
19:11
Speaker 2: for the first time, I would say, go to the
19:13
Speaker 2: calculus course on kon Academy, start watching some of those videos,
19:17
Speaker 2: do those practice And what con migo is going to
19:19
Speaker 2: be really good at is if those videos, or the
19:23
Speaker 2: practice problems, or the articles we have or the hints
19:25
Speaker 2: we provide, there's still some itchy conceptual thing you're not getting.
19:31
Speaker 2: Con Migo's really good at trying to unlock that one
19:35
Speaker 2: conceptual thing.
19:37
Speaker 1: Like what I think really kind of a narrow problem,
19:39
Speaker 1: like you're almost there, but you just need like a
19:41
Speaker 1: little one more explanation or a different kind of explanation
19:45
Speaker 1: or something.
19:46
Speaker 2: Exactly, or there's just some conceptual dimension that maybe the
19:49
Speaker 2: video didn't address that you're curious about or you want
19:52
Speaker 2: to connect it. You're learning about entropy and chemistry, and
19:54
Speaker 2: you're like, I've heard this word in computer science, how
19:57
Speaker 2: are they similar? Con Migo is great for that. I
20:00
Speaker 2: don't think conmego by itself is a place where you
20:04
Speaker 2: would just say, start being my calculus tutor for the
20:08
Speaker 2: next year year and you're gonna work through calculus with me.
20:11
Speaker 1: So that's that's where you are today. Where do you
20:14
Speaker 1: think you're going to be in a year or five years?
20:18
Speaker 2: Yeah, one year, I'm quite confident we're going to have
20:22
Speaker 2: I mean it might be six months memory. So kind
20:25
Speaker 2: of migo is going to be able to know about
20:26
Speaker 2: previous conversations. And memory isn't just even about that, it's
20:29
Speaker 2: also about being able to report back to teachers, so
20:31
Speaker 2: the teachers can say, Hey, con Migo, what have you
20:34
Speaker 2: been working on with my students? Have you noticed any
20:36
Speaker 2: general patterns amongst my students, any conceptual gaps? In fact
20:40
Speaker 2: that functionality we're already prototyped that one and we're going
20:42
Speaker 2: to launch that in the coming months, but it can
20:45
Speaker 2: also develop insights about the students, like hey, you know
20:48
Speaker 2: Mary is really into anime. Whatever I give an anime example,
20:53
Speaker 2: she lights up, or you know, Billy really likes money
20:56
Speaker 2: and so and so we're developing were already have a
20:59
Speaker 2: prototype of that where it develops these insights, but we're
21:02
Speaker 2: making it transparent to the user, like these are the
21:05
Speaker 2: insights that it's collected about you, so the.
21:07
Speaker 1: User, so people don't freak out like why does this
21:11
Speaker 1: machine know that I like anime?
21:12
Speaker 2: Or so doesn't climb the ladder of inference, which unfortunately
21:15
Speaker 2: a lots of humans do about other people, where the
21:17
Speaker 2: student can say no, actually, I just I used to
21:20
Speaker 2: be into anime. I'm not into it anymore. Or no,
21:24
Speaker 2: I know you think I really like that, but that's
21:26
Speaker 2: not you just you just went a little bit too
21:28
Speaker 2: far with that. So you're gonna have that, I think
21:30
Speaker 2: within a year. I don't know if this if we're
21:32
Speaker 2: going to have this out to hundreds of thousands or
21:34
Speaker 2: millions of people yet, but you're gonna have the ability
21:37
Speaker 2: to talk to conmigo much as you would talk to
21:41
Speaker 2: you know, your your your your Amazon or Apple devices.
21:45
Speaker 2: But it's going to be far more intelligent than those devices.
21:50
Speaker 2: It's going to know about all your context on con Academy,
21:52
Speaker 2: even though even if you're not talking to it.
21:54
Speaker 1: So you basically made a voice interface instead of a
21:56
Speaker 1: typing interface. Just to be clear, when you say talk,
21:58
Speaker 1: you mean speak yes.
21:59
Speaker 2: And I think the other thing that will surprise folks
22:02
Speaker 2: is how human like the voice will be in a year.
22:04
Speaker 2: In a year, I think a year from now, that
22:07
Speaker 2: whole loop of teachers develop assignments, creating rubrics, assigning it
22:11
Speaker 2: to students, students doing the assignment with an AI, the
22:15
Speaker 2: AI reporting back to the teacher that won. Yes, the
22:18
Speaker 2: student really did the work with me, like we worked together.
22:20
Speaker 2: It didn't just get copy and pasted from chat GBT.
22:23
Speaker 2: So that solves the hopefully addresses the cheating issue, gives
22:25
Speaker 2: students more support, and then also the AI gives a
22:28
Speaker 2: preliminary grade to the teacher. I think that whole workflow
22:31
Speaker 2: you're going to see in about a year, and you're
22:34
Speaker 2: going to see it beyond con Academy. We made an
22:36
Speaker 2: announcement a couple of weeks ago with Instructure, the people
22:39
Speaker 2: who make the you know, the biggest learning management system
22:42
Speaker 2: for k twelve and higher ed you might start seeing
22:44
Speaker 2: conmego there and other places they.
22:47
Speaker 1: Are, meaning in schools that use this particular platform.
22:51
Speaker 2: Yeah, right now, con Migo's only on con Academy's website.
22:54
Speaker 2: I think in a year you're going to see con
22:56
Speaker 2: Migo sit on other websites as well, not just on
22:59
Speaker 2: con Academy's website.
23:01
Speaker 1: That's a year which those all seem like sort of
23:05
Speaker 1: within this new universe we've learned to live in of
23:07
Speaker 1: where generative AI is. Now, what's the five year? Where's
23:11
Speaker 1: it going to be in five years?
23:12
Speaker 2: Yeah? The five years kind of wild?
23:14
Speaker 1: Yeah, the five years are so wild, right, like it
23:17
Speaker 1: we might not need you anymore in five years, right,
23:19
Speaker 1: That's part of what I was thinking about, like truly,
23:22
Speaker 1: like I don't know what is the five year.
23:25
Speaker 2: I think in five years you're going to be able
23:27
Speaker 2: to have an interview with a gen AI version of
23:30
Speaker 2: sal that will look like this interview you're doing with
23:33
Speaker 2: me right now. And for those listening, like you can
23:35
Speaker 2: see me right now. We're on a video conference, so
23:37
Speaker 2: you can actually see me. I actually think.
23:39
Speaker 1: Point that AI could teach me whatever derivatives as well
23:42
Speaker 1: as you could teach me derivatives at some point in
23:44
Speaker 1: the future. Do you have a sense of how far
23:46
Speaker 1: off that?
23:46
Speaker 2: I think. I think in the five year timeframe, it's
23:49
Speaker 2: an engineering problem more than a science problem at this point,
23:53
Speaker 2: where you just have to make the stuff fit together
23:56
Speaker 2: so it feels seamless and it feels really natural and magical.
23:58
Speaker 2: And that's what we're spending a lot of time doing
24:00
Speaker 2: so that the memory feels natural, so that how it
24:03
Speaker 2: holds you accountable feels natural.
24:05
Speaker 1: I mean, understand a little bit. It needs to understand
24:08
Speaker 1: better than understands now based on my very limited experience. No,
24:12
Speaker 1: or communicate its understanding or something.
24:14
Speaker 2: I think it's more. I think it's more of a
24:17
Speaker 2: even with GPT four, it's more about the prompting and
24:19
Speaker 2: the communication and what you're passing to what okay fair.
24:22
Speaker 2: I mean, in five years you're probably gonna have GPT
24:24
Speaker 2: five or six, and you're gonna have these other large
24:26
Speaker 2: language models, because you know you have at least five
24:28
Speaker 2: or six major groups are throwing billions at it.
24:31
Speaker 1: Will conmigo be basically redundant to GPT X whatever, GPT
24:35
Speaker 1: six and then like where do teachers fit in? Right?
24:37
Speaker 1: Those seem like two logical questions there.
24:40
Speaker 2: I've been thinking a lot about this and speaking a
24:43
Speaker 2: lot about this. I think you're going to see job
24:45
Speaker 2: disruption in a lot of places, but I do not
24:48
Speaker 2: think it's going to happen in teaching. I think you're
24:50
Speaker 2: going to I think if I told every teacher on
24:55
Speaker 2: the planet that Hey, all of a sudden, we discovered
24:58
Speaker 2: all this money, and we're going to hire three teaching
25:00
Speaker 2: assistants for every one of you to go into your
25:04
Speaker 2: classroom and they're going to help you with lesson planning.
25:06
Speaker 2: They're going to help you grade papers, are going to
25:08
Speaker 2: help you write progress supports, and while you know, while
25:10
Speaker 2: you're in the class session, they're going to circulate and
25:14
Speaker 2: answer any questions that the students have and then report
25:17
Speaker 2: back to you. I mean, I think every teacher like, finally.
25:21
Speaker 1: So that's the teacher side. What about the con academy side? Like,
25:25
Speaker 1: could AI render con Academy obsolete? Yeah?
25:29
Speaker 2: I think in five years, honestly, Jenna, I might be
25:32
Speaker 2: able to make real time videos very similar to Conacademy videos,
25:36
Speaker 2: but they feel like real time explanations. It will be
25:40
Speaker 2: that much better at potentially creating exercises and things like that.
25:43
Speaker 2: But I think it's still going to be better when
25:46
Speaker 2: it's anchored on a framework, on a scope and sees
25:48
Speaker 2: just as a teacher, a real teacher, right, A real
25:50
Speaker 2: teacher can do all of these things, but they are
25:52
Speaker 2: better when they have a curriculum, when they have textbooks,
25:54
Speaker 2: when they have con academy, when they have all of
25:56
Speaker 2: these other tools around it. I think the same thing
25:58
Speaker 2: is going to be true of Jenai for a very
26:01
Speaker 2: long time.
26:05
Speaker 1: We'll be back in a minute with the lightning round,
26:07
Speaker 1: in which I try to get Salt to reach into
26:09
Speaker 1: his days as the frontman of a heavy metal band.
26:19
Speaker 1: Let's do a lightning round. What's the first Bollywood movie
26:23
Speaker 1: I should watch? If I've never watched a Bollywood movie.
26:27
Speaker 2: You know, I mean, my standard disclaimer with all Bollywood
26:31
Speaker 2: movies are there will be moments in almost every Bollywood
26:33
Speaker 2: movies that will make you cringe. Cringe. There's gonna be
26:36
Speaker 2: a little bit overacting. People are going to be dancing
26:38
Speaker 2: at inappropriate times. But I'm your Khan, who's one of
26:44
Speaker 2: the more famous Indian actors and he also produces movies.
26:48
Speaker 2: He has this movie called Three Idiots that one I recommend,
26:52
Speaker 2: and it has some not you know to I think
26:54
Speaker 2: western sensibilities, some cringe worthy moments, but it's it's it's
26:58
Speaker 2: about education, you know, not to be too self aggrandizing,
27:02
Speaker 2: but the whole, the main story is about this guy.
27:04
Speaker 2: It's actually based loosely on a real person who he
27:09
Speaker 2: was very unsatisfied with the education system.
27:12
Speaker 1: Is it based on you? Yes? Or no?
27:14
Speaker 2: No, it's not based on me. It's not based on me.
27:16
Speaker 2: But he uh uh. He ends up eventually starting his
27:18
Speaker 2: own school. But he also ends up falling in love
27:20
Speaker 2: with the doctor and marrying her. And I'm like, that's
27:22
Speaker 2: my life. That's that's what.
27:24
Speaker 1: Uh? What was the what was the name of the
27:28
Speaker 1: heavy metal band you were in in your youth?
27:32
Speaker 2: The name was Malignancy until we realized someone else.
27:35
Speaker 1: Had that had that good name. It sounds like a
27:36
Speaker 1: heavy metal band. What can you give me a few
27:40
Speaker 1: bars of one of your songs?
27:43
Speaker 2: No, but come on, I have a certain brand now.
27:48
Speaker 2: I can't growling into a into a microphone anymore. A
27:52
Speaker 2: whole series of lyrics just went through my head, and
27:54
Speaker 2: my my podcast filter vetoed all of them.
27:58
Speaker 1: Just one phrase, one phrase. You don't even have to
28:01
Speaker 1: sing it, just say it.
28:05
Speaker 2: No, it was I was angry.
28:09
Speaker 1: Yeah, yeah, what's the last non work thing? You use
28:14
Speaker 1: chat GPT for?
28:16
Speaker 2: Oh, last non work thing? I partially used it to
28:22
Speaker 2: planification several vacations actually, and it worked.
28:26
Speaker 1: It was helpful. Yeah. So, you know, one of the
28:29
Speaker 1: things that's interesting to me looking at the arc or
28:33
Speaker 1: the ramp of con Academy from the outside is it's
28:37
Speaker 1: been clearly, like profoundly successful and kind of a darling
28:39
Speaker 1: of this sort of Bill Gates ted talk Silicon Valley
28:42
Speaker 1: universe and reasonably so rightly so. But I'm curious, like,
28:46
Speaker 1: what on the inside of building the organization was was hard?
28:50
Speaker 1: Harder than it looked.
28:54
Speaker 2: I think when you scale an organization, you know, I
28:59
Speaker 2: was a one person shop for many years, and then
29:03
Speaker 2: you start scaling. And I definitely think when we got
29:05
Speaker 2: to between between twenty people and one hundred and fifty,
29:09
Speaker 2: we had a lot of growing pains. A lot of
29:10
Speaker 2: organizations do I now see. I mean, when you're in it,
29:12
Speaker 2: you're like, is it just to us that we can't
29:14
Speaker 2: seem to But I realized that there's a when you
29:16
Speaker 2: get My old management philosophy was just get the smartest,
29:20
Speaker 2: most passionate people in the room. And then we'll figure
29:21
Speaker 2: it out. And I've now realized I'll get the smartest, passionate,
29:25
Speaker 2: most aligned folks in the room and work constantly to
29:28
Speaker 2: align around a true north. So yeah, yes, I think
29:32
Speaker 2: like many startups, when we were in a startup mode,
29:35
Speaker 2: we probably zigged and zagged more than we necessarily had to.
29:39
Speaker 2: But maybe it's part of just natural growing pains. But
29:41
Speaker 2: that's that's probably the biggest source of tension over the
29:44
Speaker 2: years of like, our mission is so big, should we
29:46
Speaker 2: just do one part of it? What about international? What
29:49
Speaker 2: about domestic? What about you know, English language arts? What
29:52
Speaker 2: about math? And so we've been pulled in so many
29:53
Speaker 2: different directions, and we violated a lot of basic business strategy.
29:58
Speaker 2: Basic business strategy would be like focus on just one thing,
30:01
Speaker 2: and I'd be like, I'm impatient, Like I'm I mean,
30:04
Speaker 2: I used to be the to your point. I used
30:06
Speaker 2: to be like the young guy on the scene, you know,
30:08
Speaker 2: with this A. I started out in my early thirties,
30:11
Speaker 2: and now I'm approaching fifty. I'm about to turn forty
30:14
Speaker 2: seven years old, and I'm like, wow, I only have
30:16
Speaker 2: like probably, if I'm lucky, I have another good twenty
30:19
Speaker 2: twenty five years. If I'm lucky and I started tooting
30:23
Speaker 2: now the nineteen years ago, like, we got to get
30:25
Speaker 2: on this. We got to do all the subjects, all
30:27
Speaker 2: the great all the countries as soon as possible. I
30:30
Speaker 2: want kind of kind of me to be around for
30:32
Speaker 2: hundreds of years, thousands of years. I read a lot
30:34
Speaker 2: of science fiction books, and so you know, even if
30:36
Speaker 2: I even if I wanted around one hundred years, two
30:38
Speaker 2: hundred years and serve this mission free world class education,
30:43
Speaker 2: how does it do that when I'm not there or
30:46
Speaker 2: when people who didn't know me are no longer there.
30:49
Speaker 2: And I don't want it to just exist. I mean,
30:51
Speaker 2: there is a success scenario where in a hundred years
30:54
Speaker 2: it's it's successful, but then it becomes it's the incumbent.
30:57
Speaker 2: It's just like every other large publisher. And that's a
31:00
Speaker 2: lame scenario either. I you know, if I was reincarnated
31:03
Speaker 2: at that time, I'd want to disrupt that incumbent. So
31:05
Speaker 2: how does it stay innovative and doesn't take itself too seriously?
31:08
Speaker 2: But it really is and hopefully at that point serving
31:11
Speaker 2: most of humanity?
31:13
Speaker 1: That is wildly ambitious.
31:16
Speaker 2: It is.
31:23
Speaker 1: Salcon is the founder and CEO of con Academy. Today's
31:27
Speaker 1: show was produced by Gabriel Hunter Chang and Edith Russlo,
31:30
Speaker 1: and it was edited by Sarah Nix and engineered by
31:33
Speaker 1: Amanda k Wong. I'm Jacob Goldstein, and we'll be back
31:37
Speaker 1: next week with another episode of What's Your Problem.