Latest / EDGE AI POD / Transforming Edge AI Education: Insights from Harvard's Dr. Vijay Janapa Reddi

Transcript

Transforming Edge AI Education: Insights from Harvard's Dr. Vijay Janapa Reddi · EDGE AI POD

• 0:32
  Speaker 1: Thank you.
• 0:33
  Speaker 1: Okay, well, I guess that was our intro.
• 0:46
  Speaker 1: We're having this morning with a live stream, so I'm coming here from Washington, where we have had a severe windstorm and no power, so I'm in my Jeep, but fortunately, jenny in Amsterdam.
• 1:05
  Speaker 1: Good morning.
• 1:06
  Speaker 1: Jenny, good afternoon.
• 1:08
  Speaker 3: Thanks, Pete.
• 1:09
  Speaker 3: And I'm glad you're safe there in the storm.
• 1:12
  Speaker 3: That sounds like a lot, but thanks for joining in and so we can talk to Dr Vijay Janapareddy today about the MLSIS book.
• 1:24
  Speaker 3: And I'm really excited to Dr Vijay Janapareddy today about the MLSYS book and I'm really excited to talk.
• 1:26
  Speaker 3: And yeah, so you do cut out a little bit in and out, pete, but if for after you talk, I'll do a quick little translation of what you said to the audience to make sure everyone understands.
• 1:38
  Speaker 3: But hello everybody and welcome.
• 1:40
  Speaker 3: Yeah, today we're going to talk to Dr Vijay Reddy from Harvard about the MLSIS book that he's published on the internet and it's open source.
• 1:52
  Speaker 3: So I'm excited to talk to him about that and his previous courses and what he's up to.
• 2:00
  Speaker 1: Sounds good.
• 2:01
  Speaker 1: Let me just bring Vijay on here and I'll actually jump off so I can save some bandwidth, and then I'll jump back on again.
• 2:09
  Speaker 1: Let's bring Vijay on.
• 2:10
  Speaker 1: Vijay, welcome.
• 2:13
  Speaker 2: Awesome Howdy everyone Nice being here.
• 2:16
  Speaker 2: Thanks for having me.
• 2:18
  Speaker 3: Thanks for joining us.
• 2:21
  Speaker 1: I'm going to let Jenny take over for a bit while I do some tech stuff here with my power.
• 2:25
  Speaker 3: Great there you power.
• 2:27
  Speaker 3: Great there you go.
• 2:29
  Speaker 3: Cool so, vijay.
• 2:32
  Speaker 3: So this will be a bit of a more of a casual conversation.
• 2:36
  Speaker 3: Today.
• 2:36
  Speaker 3: This is also the first Edge AI talk, because the TinyML Foundation has recently changed their name to the Edge AI Foundation, so we're no longer the TinyML talk and their name to the Edge AI Foundation.
• 2:46
  Speaker 3: So we're no longer the TinyML Talk and now we're the Edge AI Talk.
• 2:47
  Speaker 3: So you get the honor of doing the very first Edge AI Talk with us.
• 2:50
  Speaker 3: So it's really, really exciting.
• 2:53
  Speaker 3: I think the change of name really opens up a whole new world of things and a whole new audience as well, a bit of a, you know, from a smaller tiny email crowd to a more wider range of things, because edge is not just embedded devices.
• 3:09
  Speaker 3: It can be anything that's not connected to the internet or whatever your definition of edge AI is.
• 3:14
  Speaker 3: So yeah, I just thought we'd talk about a little bit about maybe you can talk a little bit about yourself, and then we can go into the book.
• 3:22
  Speaker 3: Let's just start there.
• 3:31
  Speaker 2: Sounds good.
• 3:32
  Speaker 2: I'm a faculty member at Harvard I think I graduated from here and I came back here because my wife loves the East Coast and she likes the four seasons.
• 3:36
  Speaker 2: I'll tell a little story which you'll probably kill me for later this evening.
• 3:38
  Speaker 2: But I was a professor at UT Austin for several years and then she wanted to come back east yes, that's where Jenny's from and so I tried to convince my wife to move west and we were out in California for my sabbatical for a few years.
• 3:55
  Speaker 2: And then I just remember one day it was a Saturday, beautiful, gorgeous day just sitting in the backyard kid was just kind of running around the backyard.
• 4:02
  Speaker 2: My wife looks up and she says, oh, how can anyone live here?
• 4:07
  Speaker 2: And I was like, uh-oh, this is not a good statement for a Saturday morning.
• 4:11
  Speaker 2: I know where this is going.
• 4:12
  Speaker 2: Her point was it's another perfect day, it's perfect every day, and I totally knew what that meant was.
• 4:18
  Speaker 2: Basically, she needs the seasons, and so well, right on that day I pretty much knew we're going to be back on the East Coast.
• 4:24
  Speaker 2: So that's what brought me back again to school, to Harvard, that's amazing.
• 4:29
  Speaker 3: I felt the same.
• 4:29
  Speaker 3: I felt the same way about Texas.
• 4:31
  Speaker 3: You only really get two seasons.
• 4:33
  Speaker 3: You get hot summer and frigid winter and there's like no in between.
• 4:37
  Speaker 3: So, yeah, I moved to Amsterdam a lot, for a lot of the same reasons.
• 4:41
  Speaker 3: It's nice to actually have four seasons, know, and when it's actually christmas, you feel the vibes, you know so.
• 4:53
  Speaker 3: So I'm uh, but I am excited to go back to texas for christmas this year and see my family.
• 4:55
  Speaker 3: But I totally uh vibe with your, with your wife there.
• 4:56
  Speaker 3: The it's just.
• 4:58
  Speaker 3: It's just so the samey all the time and it just.
• 5:01
  Speaker 3: You can't live in a place that perpetually feels like summer.
• 5:04
  Speaker 3: At least I can't yeah well, you know, but California is always there.
• 5:10
  Speaker 3: So yeah, but so in the past you have had a couple of courses previously published on other websites, like what is the difference between this new?
• 5:26
  Speaker 3: Or maybe you can talk a little bit about the overview of what this new book is and what gaps are you trying to fill with this knowledge that you didn't have in your previous courses that are still alive and very much active?
• 5:38
  Speaker 3: But what gaps are you trying to fill there and what audience are you trying to target?
• 5:42
  Speaker 3: Maybe we can talk about that.
• 5:43
  Speaker 2: Yeah, I think the way I'm going to kind of talk about it is from perspective that what's really going on with machine learning?
• 5:49
  Speaker 2: Right, obviously everybody's excited.
• 5:50
  Speaker 2: I mean, you know our foundation got rebranded into Edge AI.
• 5:54
  Speaker 2: Like there's a lot of excitement, but when you kind of open the hood and you kind of really look at it, I feel like it's the heydays of.
• 6:02
  Speaker 2: You know, machine learning is kind of having its heydays of, like computer science, if you kind of think about it.
• 6:09
  Speaker 2: Everybody wanted to program, all right.
• 6:09
  Speaker 2: So everybody started taking computer science and enrollment started skyrocketing.
• 6:11
  Speaker 2: And now everybody wants to do AI, you know, which obviously builds on top of computer science.
• 6:16
  Speaker 2: But it's like you know, everybody wants to do all these algorithm designs.
• 6:19
  Speaker 2: But the reality is that the simplest analogy I can use everybody wants to be an astronaut, nobody wants to be the rocket scientist, right, you want to build the algorithm, you want to build that model, you want to see it getting used and, yeah, that's a pie in the sky dream if you don't actually get up into space.
• 6:36
  Speaker 2: The question is, who's actually going to get you into space?
• 6:38
  Speaker 2: And I think there's a bit of this gold rush and I think it's like a little bit of naivety from the ecosystem which is kind of like, okay, a lot is to be done on that.
• 6:51
  Speaker 2: But I really think we're that's just the first wave and the second wave is kind of coming and the second wave is really going to be who are the people who are actually going to know how to build these systems and deploy them?
• 6:56
  Speaker 2: And I'm talking about deployment at the scale of massive data centers or it could be deployment the extreme edge, edge right, like the tiny amount of kind of things, like you need highly specialized engineering skills.
• 7:08
  Speaker 2: And that's really the sort of void I'm trying to fill is, yes, there's a lot of interest in the algorithms.
• 7:13
  Speaker 2: There's lots of material out there.
• 7:15
  Speaker 2: You know plenty of courses.
• 7:16
  Speaker 2: You know virtually every university is teaching.
• 7:18
  Speaker 2: You know AI, software aspects, but no one's really focusing on the machine learning, systems engineering aspect, aspect, and I think that's that gap that I'm really interested in kind of filling.
• 7:29
  Speaker 2: And so, to get to this point in the book, the book was kind of really written out of class notes and so forth that came out of these online courses, that when tiny ml was getting started you know I have the fortune of actually working with peteen on the TensorFlow like micro project back when it was a skunkworks kind of thing Back when he was kind of putting this team around, we were actually brainstorming this idea.
• 7:52
  Speaker 2: Oh, we should write material around this, because edge AI and all this stuff is unlike traditional ML, which is not like In a traditional ML.
• 8:00
  Speaker 2: You can just focus on the algorithm.
• 8:01
  Speaker 2: You often don't think so much about the system elements, like how much memory I have computer mind do I really need, like how is this actually going to work in a real-time deployment?
• 8:09
  Speaker 2: Right, and you're talking about maybe like loose latencies in the orders of, like you know, 200 milliseconds on milliseconds.
• 8:17
  Speaker 2: Like that's really not that crazy.
• 8:18
  Speaker 2: If you really look at Edge deployments, like if you want to have an autonomous vehicle, well, that real-time constraint there is pretty damning.
• 8:25
  Speaker 2: That really requires people who understand a little bit of that language about machine learning from a theoretical sort of perspective, but also understand the engineering aspects, and I think that's that really unique niche that I'm really super passionate about.
• 8:40
  Speaker 3: So do you find, like, with all the hype with like chat-GBT and stuff and these big LLMs and transformer models, you do find that people who get people get excited, and maybe even students they get really excited about these big models?
• 8:54
  Speaker 3: And like, even in industry, do you find that people are not really paying attention to optimizing for smaller hardware, because I mean, in this world we're finding these big warehouses full of GPUs that you don't need to optimize if you can have unlimited compute.
• 9:11
  Speaker 3: So this course is primarily trying to focus on showing the world that this is what needs to be done for the future of AI.
• 9:23
  Speaker 2: Originally I did start with that sort of a mindset.
• 9:25
  Speaker 2: Like you know, I was kind of teaching these embedded ML courses and at Harvard and online, you know, I was teaching this tiny ML courses, but when I started really just kind of, you know, taking my class notes and just trying to scribe them out.
• 9:38
  Speaker 2: In all honesty, you know, this online book is still very much a work in progress.
• 9:42
  Speaker 2: It's it's not a textbook yet.
• 9:43
  Speaker 2: It's in its transition phase right now and I would say more, like you know, we're done with all the things that I have in my head about what I think students need to know.
• 9:51
  Speaker 2: And as I started, writing it originally it was literally called Tiny Machine Learning Systems kind of thing, because it's just class notes from my book, and then I started writing it down.
• 9:59
  Speaker 2: Then I started realizing something quite critical.
• 10:01
  Speaker 2: I was like it didn't matter Sometimes some of the concepts I was talking about.
• 10:04
  Speaker 2: It didn't matter if it was a big system or a small system.
• 10:07
  Speaker 2: It's like an operating system, right, if you go back to a conventional sense, you kind of think about it.
• 10:11
  Speaker 2: Almost every single one of us has taken a computer organization course or an operating systems course where you really teach the fundamentals, right, let's say, if it's an operating systems course and you learn about scheduling, you talk about virtual memory and memory management principles and all these you know, device management and so forth.
• 10:32
  Speaker 2: Those are core principles.
• 10:34
  Speaker 2: Now, it doesn't matter if it's like you know, it's a big operating system.
• 10:36
  Speaker 2: Yeah, it's a big operating system.
• 10:38
  Speaker 2: Managing something at the scale of, like you know, a hyperscaler, right, managing the.
• 10:41
  Speaker 2: A hyperscaler, right, managing the clusters.
• 10:42
  Speaker 2: Yeah, it still has to do scheduling.
• 10:44
  Speaker 2: Yes, it has to do memory management and all of this.
• 10:46
  Speaker 2: Just the numbers change.
• 10:47
  Speaker 2: You go to an embedded operating system same thing, you still have to do scheduling, and the mechanisms, the way you implement them change, but the fundamentals stay the same, and so the sense that I'm really getting at is like it's not whether it's a big ML system or a tiny ML system, it's really just an ML system, and that is what I myself personally kind of discovered.
• 11:06
  Speaker 2: I was like, oh, that is when the light bulb really went off in my head, which I was like, oh my gosh, this can be thought at a massive scale and should be thought at a massive scale, like we teach one-on-one digital logic design or any of those kinds of courses.
• 11:20
  Speaker 2: So that was sort of an eye-opening moment for me, and that's when I really started thinking okay, it's just machine learning systems.
• 11:29
  Speaker 2: Now, of course, there are advanced concepts to the points that you're talking about, which is like okay, if I'm going to build a hyperscaler, I need to think about massive distributed interconnects and so forth that I need to have.
• 11:39
  Speaker 2: But that's an advanced concept.
• 11:43
  Speaker 2: And I feel like most of the time, like when you look at institutions that do teach these things, like my school or you know my colleagues at other schools, we all already know all this material and so, in some capacity, we already know it's in our head, so we teach them in our classes yeah but when you're talking about it, trying to make it like a universal kind of a curriculum that you should just know about, right, just like you teach python or just like you teach c and so forth then you really want to get into what are the fundamental principles that everybody needs to know yeah that's kind of really been the fuel that's been kind of driving me and getting feedback, you know, in terms of what needs to be covered, and so that's the way I've been approaching what we should put in the book.
• 12:29
  Speaker 3: That sounds definitely aligns with my own education at UT, because the electrical engineering program is very much a bottom-up approach where you start from binary and you go all the way to Java, so definitely feels like it would definitely fit into that education.
• 12:40
  Speaker 3: They're probably already having a course along those lines at UT Austin as well, so that totally makes sense.
• 12:45
  Speaker 3: So I guess that segues into my next question.
• 12:48
  Speaker 3: So you told us that you saw, ml systems engineering is something that definitely needs to have a gap filled.
• 12:58
  Speaker 3: But who is this online book really designed for?
• 13:01
  Speaker 3: Is it for beginners who are just trying to get started out in AI?
• 13:05
  Speaker 3: Is it people who are interested in embedded engineering?
• 13:08
  Speaker 3: Is it people who are interested in generative AI?
• 13:10
  Speaker 3: I know your book touches on all those topics but, like, who is like the ideal person to get started?
• 13:16
  Speaker 3: What prerequisites would they need in order to get started with this book?
• 13:20
  Speaker 2: Yeah, it's a work in progress, as I've said, and so, keeping that in mind, I was just talking to the publisher yesterday was kind of helping me sort through.
• 13:29
  Speaker 2: Okay, who's going to be the audience, right?
• 13:30
  Speaker 2: Yeah, you're asking and I I'm really right now focused on if this was to be thought at an undergraduate level as core courses you need to have as part of your package of understanding how you build computer systems in the future.
• 13:44
  Speaker 2: What are the things to talk about?
• 13:46
  Speaker 2: And so that's where it's been really kind of targeted that like junior senior year students, 30 or fourth year students or intro graduate level students who understand this.
• 13:57
  Speaker 2: I've also tried to cater it for people who understand a little bit about, uh, machine learning but they want to understand the system implications.
• 14:03
  Speaker 2: All right, um, so, for example, like you know, people will come up with the model.
• 14:07
  Speaker 2: But, okay, what if I want to trade off, you know, accuracy versus safety or accuracy versus you know some performance, like, okay, well then, what sort of optimizations are these systems people talking about?
• 14:16
  Speaker 2: What does it mean to say, do unstructured pruning?
• 14:19
  Speaker 2: yeah people might have heard about pruning, but often we talk about structured, unstructured and so forth, so try to kind of cater it for a bunch of different people, and so one of the things that happened just recently was that I ended up in the about the book section.
• 14:31
  Speaker 2: I ended up actually adding the section about like okay, depending on which persona you are, if you are a tiny ml newbie or you're just an ml systems person, or you want to be a full system, you know full ML expert, you know full stack expert.
• 14:45
  Speaker 2: And so I kind of wrote out like a little roadmap of these are the chapters to touch.
• 14:50
  Speaker 3: Yeah, that's awesome.
• 14:51
  Speaker 3: Okay, so it's junior senior students, early graduate, but it could also be people who are just trying to advance their knowledge of these other areas of ml systems in industry as well yeah, yeah, absolutely yeah.
• 15:06
  Speaker 2: I I honestly think at this point it'll probably be a lot easier, candidly, for industry people to kind of be able to pick it up, because when I started writing it I was just kind of like okay, these are the core concepts to kind of get through.
• 15:15
  Speaker 2: But that kind of implicitly assumes that you have a little bit of context about what's a microcontroller.
• 15:20
  Speaker 2: There's students who kind of go through it and they're like you just kind of talk about these microcontrollers, the system's like, like you know, it's kind of like second nature yeah now an industry practitioner would totally would not think twice about that word and probably just gonna move on and be like, of course I know what that is roughly um.
• 15:35
  Speaker 2: So that's where, in fact, like I think it's much more accessible right now for industry people or people who have just been playing around generally in space and want to pick pickup ml in fact, yeah, transitioning over to the students, actually being a faculty member, that's where my head is most of the time.
• 15:50
  Speaker 2: It's almost like a higher bar, in fact, you know, and so that's where I'm really spending a lot of my time is trying to lower that material down.
• 15:57
  Speaker 2: In a way students can progress from one class to the other.
• 16:00
  Speaker 2: So, yeah, glad you kind of pointed that out.
• 16:04
  Speaker 3: Yeah, we think we talk about that a lot at Edge Impulse, especially with our documentation, where there's so many people coming from so many different angles, personas of the documentation is such a big thing Because, yeah, you're totally right, there's completely different routes for people who don't know what a microcontroller is versus what is they already know or they already have a microcontroller that they want to deploy onto.
• 16:26
  Speaker 3: So those are completely two different knowledge ranges.
• 16:28
  Speaker 3: So it completely changes the routes and what you need to include in the book.
• 16:33
  Speaker 3: But again, like there's almost an infinite combinations of different people's knowledge versus what you can include into a book, so you really have to like, choose, like what are the most important pieces.
• 16:42
  Speaker 3: So I'm sure you and your publisher are gonna work really hard on what is going to be the most important thing.
• 16:48
  Speaker 3: So yeah I'm looking forward to that.
• 16:50
  Speaker 3: All right, so you mentioned publishing.
• 16:52
  Speaker 3: Will you?
• 16:53
  Speaker 2: will this go on like amazon, or well, um, I'm working with a publisher to try and convert this into a hard copy, because people are looking for it, although I think the field is still moving.
• 17:04
  Speaker 2: So a hard copy is like okay, I mean I have hard copies, but each of these books is it's a hard copy, not because I opened the book, so much because when I was a student, like it dramatically influenced my thinking process and that's why I have like a handful of books, pretty much because if you look at the books you'll know how I tend to think yeah, yeah I think like it was interesting just because yesterday I got an email from someone saying like I really want to have you know um a published copy of the book, and I was like oh, it's not ready.
• 17:32
  Speaker 2: But they were like no, I still would love to be able to have it.
• 17:34
  Speaker 2: And I was like yeah why?
• 17:36
  Speaker 2: And it's like oh, it's because it's it reminds me of things, and so it's almost like yeah, yeah, like a visual cue.
• 17:43
  Speaker 3: Visual cue to like this is what you're interested in today, like robotics, ai, da, da, da, da, da Like.
• 17:48
  Speaker 3: Yeah, I find that with people who have my book too, like I'm like, why are you carrying around this huge book when you can get the PDF?
• 17:55
  Speaker 3: And they're like well, because it's just, it's nice to have the physical knowledge in your hand.
• 17:59
  Speaker 3: You know so.
• 18:00
  Speaker 3: So I totally get that.
• 18:02
  Speaker 3: So we have actually a question from the audience.
• 18:05
  Speaker 3: Let me read it to you.
• 18:06
  Speaker 3: I can't pull it up on the screen because I don't have the same powers magical powers as Pete.
• 18:13
  Speaker 3: But Zaya from LinkedIn says thanks, vijay, for the perspective.
• 18:16
  Speaker 3: How do you approach the tools, hardware and software for your ML systems course?
• 18:21
  Speaker 2: Yeah, that's a good question.
• 18:22
  Speaker 2: I'm super fortunate to be working with a couple of people who are very passionate about, like you know, helping out, and so the book is really not just me in my head.
• 18:30
  Speaker 2: It's, like you know, a lot of people actually communicate with offline and one of the things the beautiful aspect of that hands-on component there's a lab section in there and those lab sections have all been contributed by a colleague, marcelo, who's actually quite well known in the TinyML space.
• 18:47
  Speaker 2: Yeah, marcelo Robai.
• 18:48
  Speaker 2: He's the one who's actually contributed all of the lab stuff and it's been great because, you know, we have this TinyML 40 effort that Marco and Brian help lead and Marcelo is sort of like is critical in that piece and he's been doing lots of workshops and so one of the things that we did was like we took that labs portion, which is really kind of you know the hands on component, right, and I think that's one of the fascinating things about you know the systems kind of thing, because the question is like you know, think about it, if you do ml.
• 19:16
  Speaker 2: I'll start by asking students how many of you have actually built a ml data set?
• 19:22
  Speaker 2: the answer is almost zero zero yeah if you want to build a big model, you can't.
• 19:26
  Speaker 2: I mean, if you have a decent sized model, you immediately need like a million images.
• 19:29
  Speaker 2: You get that right.
• 19:31
  Speaker 2: So by and large, you just download someone else.
• 19:33
  Speaker 2: But data is the way you program these systems.
• 19:36
  Speaker 2: So if you don't understand how to actually program the system through the data collection process, then it's pretty much like you know you're just playing around with a black box.
• 19:44
  Speaker 2: That's why everybody calls it black box.
• 19:45
  Speaker 2: It's not as bad a black box as people make it seem.
• 19:49
  Speaker 2: That's just because you avoid the piece that you actually program it with.
• 19:52
  Speaker 2: So if you actually understand the data side, you know what's going on.
• 19:55
  Speaker 2: And that's what makes, like, the whole tiny ML piece super interesting, because in Marshal's labs it starts off with the whole data pipeline.
• 20:03
  Speaker 2: You're collecting data, you're deploying ML on these tiny ML systems, which is very accessible.
• 20:09
  Speaker 2: Then suddenly you only need 30, 40 examples, ninety percent of the time.
• 20:12
  Speaker 2: I honestly like that.
• 20:13
  Speaker 2: It's actually all based off of Edge and now that I think about it, um, because it's very accessible for students to be able to, like, just through a gui, do that and you still get the fundamental elements.
• 20:25
  Speaker 2: Um, yeah, you know.
• 20:25
  Speaker 2: Yeah, ideally you take the course and then of course you're understanding what edge impulse is doing behind the you know, behind the scenes and so, yeah, a lot of that hands-on component is really based on, like you know, what marcelo kind of put together and we've been very systematic about it and trying to make sure we cover different modalities of sensory input coming in.
• 20:43
  Speaker 2: Just you know, different kinds of input data, because you have to deal with them differently and very structured and making sure when we support it on different platforms, that the same set of labs are actually supported on different platforms.
• 20:54
  Speaker 2: Yeah Right, so so that's sort of a thing.
• 20:56
  Speaker 2: Supported on different platforms yeah right, so so that's sort of a thing.
• 20:58
  Speaker 2: So then there's a little bit of portability, and this is why we don't support any arbitrary thing, and through that we've been trying to kind of curate all the material into one cohesive thing.
• 21:04
  Speaker 2: Um, so yeah, so that's kind of how we approach the hardware software interface it's so.
• 21:10
  Speaker 3: It's so funny what you said about the data collection, because we talk about this a lot at the the previous tiny ml foundation summits, now the edge ai foundation um conferences, um, but we talk about this a lot at the previous TinyML Foundation summits, now the Edge AI Foundation conferences, but we talk about this a lot that people are just not interested in collecting data sets because of all the massive ones that are already out there.
• 21:26
  Speaker 3: But I find at Edge Impulse like because we have such an easy flow and other AI tools also have easy flows but of course I'm biased for Edge Impulse I find that the data collection part is some of the most important and most fun part and I also think it's some of the most relevant part for actual industry engineering for edge AI applications.
• 21:47
  Speaker 3: Because once you get into a company where they're actually doing AI for various automobile sensors or AI for window detecting your window open and closed and burglaries and security, there's not going to be like a preset data set on the internet available for you that works exactly for your use case.
• 22:10
  Speaker 3: So it's like it's a really important skill in order to think about, like, how do you orient?
• 22:14
  Speaker 3: You, know your different ML classes, how do you, how do you collect the data accurately and securely?
• 22:19
  Speaker 3: How do you get it over the air?
• 22:21
  Speaker 3: How do you store it on the device?
• 22:23
  Speaker 3: So I think that's actually one of the most fun parts, and we find at Edge Emplos, too, that it's one of the most crucial parts.
• 22:30
  Speaker 3: So it's really it's heartening to hear you talk about that and I think about it like.
• 22:37
  Speaker 2: people make this mistake when they take ML courses and so forth.
• 22:39
  Speaker 2: They think data science is actually data engineering.
• 22:41
  Speaker 2: There's a big difference between the two.
• 22:42
  Speaker 2: It's a fool's thing to think just because you know data science it translates to data engineering A lot of the things that Jenny just mentioned, which is how you manage that data and how you think about what aspects to focus on, and so forth.
• 22:54
  Speaker 2: It's not necessarily data science.
• 22:56
  Speaker 2: Data science is more the curation part, the think about what aspects to focus on, so forth.
• 22:58
  Speaker 2: It's not necessarily data science.
• 22:59
  Speaker 2: Data science is more of the curation part.
• 22:59
  Speaker 2: The engineering part is a little different.
• 23:00
  Speaker 2: This is where, again, it goes back to that engineering element that's kind of overlooked so much and in practice, I think this is why there are crazy stats on the internet that basically say that 87% 90% of industry experiments on AI trying to kick off an ai group failed because that's data science project.
• 23:20
  Speaker 2: it's not a machine learning systems project yeah what they're really doing is just toying around with it and they're like oh, I got a proof of concept.
• 23:26
  Speaker 2: I'm like, yeah, that that doesn't work.
• 23:27
  Speaker 2: You have, it takes a massive new team of people who understand this lingo to pull it off yes yeah, there's a lot of this misconception or like this gap in what's actually out there and what's actually needed versus, like you know, what people mostly talk about.
• 23:44
  Speaker 3: Yeah, well, people are interested in what's the most flashiest thing.
• 23:47
  Speaker 3: You know so.
• 23:47
  Speaker 3: But yeah, data engineering is arguably one of the most important parts of ML systems engineering.
• 23:54
  Speaker 3: But we have another question from the audience for you, and it's a simple one what microcontrollers are used in the labs important parts of ML systems engineering?
• 23:57
  Speaker 3: But we have another question from the audience for you, and it's a simple one what microcontrollers are used in the labs?
• 24:02
  Speaker 2: Oh yeah, so we use.
• 24:03
  Speaker 2: We try to grade them intentionally by cost, just because we've learned that different people have different levels depending on where you are in the world.
• 24:14
  Speaker 2: A dollar goes a long way or a very short way.
• 24:17
  Speaker 2: So, um, the simplest ones that we have are like the, the shower sense.
• 24:22
  Speaker 2: Um, seed studio has been a really good supporter of us, have donated many kids, and so the shower sense is something that we use um, because that's only about 15 bucks.
• 24:30
  Speaker 2: Still in some places that can still be a lot.
• 24:32
  Speaker 2: Um, I'm talking about devlet countries and so forth, where we do outreach.
• 24:35
  Speaker 2: Um.
• 24:36
  Speaker 2: Then we have the next one.
• 24:39
  Speaker 2: Up is a pretty beefy system which is like the Nikola vision from the Arduino, which is more of like a professional grade embedded microcontroller embedded system.
• 24:47
  Speaker 2: And then, of course, it goes up to.
• 24:49
  Speaker 2: Marcelo just recently pushed up the RASPi support In that one of the cool things that he actually added in was generatively high at the edge.
• 24:58
  Speaker 2: So you know oh yeah, so I'm like a small language model.
• 25:01
  Speaker 2: It was one of the most recent updates, so just much like the tiny foundation has been expanding its scope, like one of the things in the labs we have intentionally been expanding is going from microcontrollers to a decent scaled system.
• 25:13
  Speaker 2: Um, and this is not like I'm thinking that we'd be tracking what is happening in industry.
• 25:19
  Speaker 2: It's more that we tend to think about it from like a very pedagogical perspective, like what is going to be accessible for students at scale, at university level, and what are those course in a box packages that we can actually put out.
• 25:33
  Speaker 3: And I find, especially with the microcontrollers and the boards that you've chosen for the book, they are like like it's just one funnel with lots of documentation, but it's also the foundations and fundamentals that you learn from programming for the ESP32 or for the Raspberry Pi.
• 25:50
  Speaker 3: Those fundamentals that you learned for that board apply to almost every other microcontroller that you can find as well.
• 25:56
  Speaker 3: You know you could get a completely different board and as long as you understand, you know how to compile the code and how do you, you know, incorporate new sensor drivers.
• 26:05
  Speaker 3: The fundamentals are there from what you've already learned for the ESP32 that you can apply towards, you know, the NVIDIA Jetson or something.
• 26:13
  Speaker 3: So I think those are really good.
• 26:15
  Speaker 3: Especially those boards are so well documented and they're so widely used.
• 26:19
  Speaker 3: You haven't any sort of like compile errors.
• 26:21
  Speaker 3: You can just literally copy and paste it into Google, and someone has already experienced it before.
• 26:26
  Speaker 2: ChatGPT.
• 26:28
  Speaker 3: Yeah, chatgpt can help you.
• 26:30
  Speaker 3: Like, do it now, like I don't even think.
• 26:32
  Speaker 3: I don't even think of those terms just because I didn't go through school with ChatG, bts.
• 26:36
  Speaker 3: I'm always thinking like Google, google, google.
• 26:39
  Speaker 3: But you're right, now it's chat to BT and even the chat to BT web browser.
• 26:44
  Speaker 3: Yeah, that's right.
• 26:46
  Speaker 2: Yeah, that's an amazing time right now.
• 26:50
  Speaker 3: So we have another question from Cesar.
• 26:54
  Speaker 3: Jaxx is now supported by LiteRT, formerly TensorFlow Lite, offering a simple and efficient option for tiny ML implementations.
• 27:02
  Speaker 3: The ML workflow would be highly simplified for beginner users.
• 27:06
  Speaker 3: I guess the question here would be have you considered talking about JAX in the book or in your courses?
• 27:13
  Speaker 2: Yeah, definitely, yeah, people have definitely given me some really good feedback in terms of, like the AI framework section, for instance, to be to be, you know, very blunt and transparent.
• 27:21
  Speaker 2: It was like it's very biased towards TensorFlow ecosystem.
• 27:24
  Speaker 3: Yeah.
• 27:24
  Speaker 2: And honestly it kind of comes from just you know, I know the TF ecosystem really well inside out and so when you know again, these were coming from my class notes and when I was talking about it I like to be able to cover the full spectrum of the stack and help people understand, okay, what's the difference between intensive flow versus tens of flow, light versus tens of flow like micro or like you know.
• 27:40
  Speaker 2: And so one of the criticisms there is, like it's a very tf based kind of thing and the two pieces that people have asked is, like you know stuff around jackson as well as, like you know, around pytorch, because pytorch is now trying to starting to spam that ecosystem, and so we're actually actively working on pulling in jackson, pytorch into the into the curriculum pieces.
• 27:57
  Speaker 2: It's not there and honestly probably won't be there until I would say like March April is earliest.
• 28:03
  Speaker 3: I would think that we would actually push some new updates would this be an opportunity for someone like in the community to provide some, you know, like a GitHub issue or GitHub PR, to add some content?
• 28:16
  Speaker 2: yeah, absolutely yeah, that's a key thing that Jenny just kind of touched on is like well, you know, when we started doing this book thing, it's it is an open source project and one of the key things like if you go to the, the mlsbookai, you'll see that there's an acknowledgements kind of tab on the left side, right at the top, and it's because you know there are contributions that come in and I don't care if it's a contribution is like oh, here's a major section with some crazy technical depth, or if it's like, hey, I found these typos and you know these mistakes that people caught, I give kudos to everybody.
• 28:43
  Speaker 2: And so if anybody who just kind of issues any sort of issue and you know files an issue and you know gives a patch of any sort, it automatically pulls that person up into the acknowledgements thing.
• 28:52
  Speaker 2: So I'm trying to very much kind of build it around the open source kind of a thing.
• 28:56
  Speaker 2: So the idea yeah, jenny, thanks for bringing it up is how do we kind of create this learning more from not just one individual, because we all know more than you know collectively.
• 29:06
  Speaker 2: We know so much more.
• 29:07
  Speaker 2: So if you have thoughts or ideas and want to just, you know, work on something just ping me or just kind of you know, follow it or work on it.
• 29:15
  Speaker 3: I mean there might be a little incentive here to like really get on it, because if you do it soon you might be mentioned in the hard copy, right.
• 29:22
  Speaker 2: Oh, yeah, that's true.
• 29:23
  Speaker 2: You're good, I didn't even think about that.
• 29:26
  Speaker 2: Yeah, that's true.
• 29:31
  Speaker 3: So everybody who's watching get out there and submit your GitHub issues and your PRs and have the team at Harvard review them.
• 29:38
  Speaker 3: Then you can be in a book.
• 29:41
  Speaker 2: Yeah, yeah, it's been awesome, like you know, a good number of contributions A lot of them are actually Harvard students and so forth that I teach.
• 29:50
  Speaker 2: When I'm teaching the class, I also get the students to kind of contribute back in, like, mostly they contribute in the context of like oh, here's a seminal paper.
• 29:57
  Speaker 2: So this is one thing that we focus a lot on in the book, which is a lot of what's in there is really built on fundamental research.
• 30:06
  Speaker 2: So when it's fundamental research, I try to point people in the writing to be like oh yeah, here's the paper that we're actually building off of, because this was a seminal paper, and I think that's actually very important for students and practitioners to kind of learn, because it tells you about which conferences people are publishing in and just kind of.
• 30:25
  Speaker 2: You have to know that language, because I think ML is so different than classic space where things are still evolving, so you need to know where to go and look for the latest information.
• 30:33
  Speaker 2: And that's where, like later on, perhaps, we can touch on that AI bot that we created, because, partly, we're working on these kinds of things.
• 30:41
  Speaker 2: How do we keep things updated automatically?
• 30:44
  Speaker 3: Yeah, very interesting.
• 30:45
  Speaker 3: Ok, so another question, let's see From Emila.
• 30:55
  Speaker 3: Thank you.
• 30:56
  Speaker 3: I wonder if in the lab DSP special features also the pre-processing FFT wavelet windowing is analyzed from the hardware resources point of view.
• 31:06
  Speaker 3: Or is FFT probably too heavy for microcontrollers?
• 31:11
  Speaker 2: Yeah, so that's a good point.
• 31:12
  Speaker 2: Yeah, there is a dedicated, you know, general lab, independent platform, independent sort of you know section where we talk about feature engineering and so forth and so, yes, so some of that preprocessing one could argue is sort of heavy lift.
• 31:27
  Speaker 2: But again, you know, it really depends upon what is your quote unquote microcontroller right?
• 31:33
  Speaker 2: I mean these microcontrollers are getting fairly advanced because ARM is obviously caught on to this notion that, okay, well, ml is going to be at the extreme endpoint.
• 31:40
  Speaker 2: So what might be today might be a bit of a heavy lift.
• 31:45
  Speaker 2: I don't necessarily see that being a heavy lift in the future.
• 31:48
  Speaker 3: Yeah.
• 31:49
  Speaker 2: Because it's just become obvious that, okay, now we're going to be running all this kind of code and it needs some specialized engine.
• 31:54
  Speaker 2: So I would think more about it from a how do you future proof the learner?
• 32:01
  Speaker 3: um, again, that's where we're coming out from yeah, I mean dsp is also a fundamental thing.
• 32:07
  Speaker 3: That is is really key to understanding for especially optimization optimization of ai for edge devices and resource constrained devices.
• 32:15
  Speaker 3: So that's really important.
• 32:17
  Speaker 3: But, as I say in all my webinars with Edge Impulse, what is true today will not be true within at least two years.
• 32:24
  Speaker 3: So it's really exciting to be in a field that's constantly evolving and changing so rapidly, but it does make it such that you know the courseware, the books.
• 32:35
  Speaker 3: They constantly have to be updated.
• 32:37
  Speaker 2: So one of the things that I dread about such that you know the courseware, the books they constantly have to be updated One of the things that I dread about making it into a hard copy.
• 32:42
  Speaker 2: To be honest, I think I honestly have it easier than, like you know, the stuff that, for instance, jenny's book, or stuff that Pete Worden had originally written with Dan, because in those books you know you have to, you're driving, you're teaching through example, which is a very powerful concept, but the code changes so fast.
• 33:00
  Speaker 2: Yeah, and I buy those books even though I know you know the code won't run right, but I still get the purpose of having them on the shelf.
• 33:07
  Speaker 2: That's where I kind of like the online aspect of you know, having keeping things online and so forth.
• 33:15
  Speaker 3: Yeah, no, I completely agree.
• 33:17
  Speaker 3: If I was ever to do another book, I would want to do it the way you've done it as well.
• 33:21
  Speaker 3: But because me and Dan we're going to be by the way, my book is AI at the Edge, just not to plug it, but that's the book that I'm talking about.
• 33:28
  Speaker 3: But next summer Dan and I are planning to do to be updated.
• 33:35
  Speaker 3: Every single line of code does not work anymore.
• 33:39
  Speaker 3: So it would be a lot easier if we could just do a GitHub PR.
• 33:42
  Speaker 3: But you know it's all about.
• 33:45
  Speaker 3: GitHub what.
• 33:48
  Speaker 2: People do like that hard copy element.
• 33:50
  Speaker 2: That's what I was learning they do.
• 33:52
  Speaker 3: I like it too.
• 33:53
  Speaker 3: Not going to lie, it's really fun to be able to call yourself a public.
• 33:58
  Speaker 1: Yeah.
• 34:01
  Speaker 3: But yeah, no, we have another question from Zayed.
• 34:05
  Speaker 3: One challenging aspect of Edge AI is the variety of domain expertise.
• 34:10
  Speaker 3: I find it challenging to provide this expertise in an engineering course.
• 34:14
  Speaker 3: Any suggestions on how to create this balance in a course?
• 34:19
  Speaker 2: Yeah, that's a good question.
• 34:20
  Speaker 2: I'm just thinking of my head side.
• 34:22
  Speaker 3: No, take your time.
• 34:23
  Speaker 2: Aspect of Edge AI is the variety of domain expertise.
• 34:28
  Speaker 2: Challenge you to provide this expertise in an engineering course.
• 34:34
  Speaker 2: Let me try.
• 34:38
  Speaker 2: I feel like we've thought about this.
• 34:40
  Speaker 2: Like in teaching these courses, the labs specifically focus on different modalities, like we have.
• 34:50
  Speaker 2: We have a computer vision thing, we have an audio signal kind of a thing, and then we have a theories kind of a thing.
• 34:55
  Speaker 2: You know, I am new kind of thing and that's always been the three.
• 34:59
  Speaker 2: I think this dates back way back, even when I was talking to Pete, like when we're working on the original frameworks and stuff about what are some good examples to actually have.
• 35:07
  Speaker 2: Yeah, I think largely the community has kind of landed on these modalities because you're right that the domains where these things are deployed are highly specialized, but the modality is not that varied in some capacity, buried in some capacity and to that end, while the three might not be comprehensive, uh you know video signal, audio signal and telemetry kind of input, they do form the basis and that's the way at least you know marcello myself, like you know a couple others that I know we all approach it and say okay yeah
• 35:40
  Speaker 2: that's what, that's how we explain the labs and that's why, pretty much when I teach my own course, I kind of of structured around that I didn't say, okay, they're going to be three labs plus your project, and that's the way to kind of, you know, teach that.
• 35:50
  Speaker 2: Okay, well, this particular modality has all these use cases and you know so.
• 35:55
  Speaker 2: But we'll just take one slice of it.
• 35:56
  Speaker 2: Jenny, are you there?
• 36:02
  Speaker 2: You're frozen, to me at least.
• 36:03
  Speaker 3: Sorry, I'm back.
• 36:04
  Speaker 3: I think I'm back.
• 36:06
  Speaker 2: Sorry, yep.
• 36:17
  Speaker 1: And you're frozen again.
• 36:19
  Speaker 1: Hello, howdy Pete, I'm coming in.
• 36:21
  Speaker 1: Oh, here we go.
• 36:21
  Speaker 1: I thought maybe Jenny was frozen.
• 36:22
  Speaker 1: Oh, she might be frozen, frozen, yeah.
• 36:29
  Speaker 1: So I'm just just to give people context.
• 36:32
  Speaker 1: Didn't tune in in the beginning.
• 36:33
  Speaker 1: I'm in from Bellevue, washington, and we had the kind of while he's in Boston terms, we call it a wicked, wicked bad winter storm here yesterday.
• 36:43
  Speaker 1: So we lost power.
• 36:44
  Speaker 1: So I'm actually in my Jeep, I'm parked at the Starbucks Doing a live stream.
• 36:49
  Speaker 1: I'm living the dream I saw I was backstage and I saw Jenny was frozen, so I thought I would jump.
• 36:56
  Speaker 3: You're good.
• 36:57
  Speaker 3: So I have this weird setup paid at my house where I have a literally like a 50 foot ethernet cable running through my I have I literally like a 50 foot ethernet cable running through my entire house.
• 37:06
  Speaker 3: What?
• 37:07
  Speaker 3: happens is like one of the pets will like, and then it's not the best solution, you know, but it's an engineered solution.
• 37:16
  Speaker 3: So apologies for that.
• 37:18
  Speaker 3: But I did have one comment on the last question we were talking about and we encounter this question a lot at Edge Impulse with our solution engineers.
• 37:27
  Speaker 3: We don't have the domain expertise for every single domain in the world, you know.
• 37:32
  Speaker 3: So that's why when we're talking to our customers and people that we want to engage with with the Edge Impulse platform, we always emphasize that they should be well, they should be, or they likely are the expert of their own domain, of their own data set of their own, what they want to do with their product.
• 37:50
  Speaker 3: And if they don't have that expertise, it's usually like then it's usually time to like bring in another person or bring in a different solution engineer that already has this experience before.
• 38:03
  Speaker 3: So I think it's like you said.
• 38:05
  Speaker 3: It's a focus on the fundamentals of domain expertise, or fundamentals of various types of sensors, computer vision, time-based frequency.
• 38:16
  Speaker 3: So focus on learning how to deal with those fundamentals and then extracting more information about your specific domain once you get into industry.
• 38:25
  Speaker 3: So I just think it's impossible to teach about every single type of sensor or use case out there in a course.
• 38:33
  Speaker 3: I think, yeah, teaching the fundamentals is more important, for sure.
• 38:39
  Speaker 2: Yeah.
• 38:42
  Speaker 1: Yeah, you had mentioned about this kind of AI tutor or AI powered tutor.
• 38:48
  Speaker 1: Is that part of the book itself, or is that a side project that you're just integrating, or what's that all?
• 38:54
  Speaker 2: about.
• 38:54
  Speaker 2: Yeah, so as context for people listening in, one of the things obviously that's happening is these LLMs are mighty powerful.
• 39:02
  Speaker 2: Obviously, we interact with them through ChatGPT and all that good stuff, and the thing that I've been very passionate about is kind of like this notion, which I'm gonna call generative learning, sort of this fusion between generative AI technologies and this traditional pedagogical learning, and how do you kind of bring these two together?
• 39:18
  Speaker 2: And when you think about a book, one of the things that would always drive me nuts whenever I'm looking at any book and you know, having been a student too is like textbooks are very static in some sense.
• 39:29
  Speaker 2: Right, you know, there are amazing textbooks because people do spend an incredible amount of effort, you know, trying to write every single word very thoughtfully.
• 39:36
  Speaker 2: But even when you do that, there are different learners to the point that Jenny was bringing up earlier learners to the point that Jenny was bringing up earlier.
• 39:43
  Speaker 2: There are people who are completely new.
• 39:44
  Speaker 2: The people who are, like you know, come in from some of the classes especially.
• 39:46
  Speaker 2: They go to university, like Harvard, for instance, where we really value liberal arts education and kind of mixing things up, like we take a lot of pride in that, like intentionally mixing things up.
• 39:55
  Speaker 2: But when you have that sort of a mix, you have students who come in from many different backgrounds and it's never really quite uniform, and so how do you sort of level set that right and make sure that education is sort of personalized to an extent?
• 40:06
  Speaker 2: And what generative technologies.
• 40:09
  Speaker 2: I think it opens up this whole new space of being able to let the learner learn the material in a way that is accessible to him or her, and that's where we have this um generative bot.
• 40:23
  Speaker 2: Can I share my screen?
• 40:24
  Speaker 2: I don't know if it'll work, but maybe that'll be useful.
• 40:27
  Speaker 1: Yeah, if you hit the present button down there, that could work.
• 40:30
  Speaker 1: Let's bring you back to the.
• 40:33
  Speaker 1: We're in this mode.
• 40:36
  Speaker 1: Hello, look out.
• 40:38
  Speaker 2: Okay, great, so, yeah, so can you see my screen?
• 40:41
  Speaker 2: Yes, we can see your screen, Okay, yeah.
• 40:43
  Speaker 3: So can you see my screen?
• 40:45
  Speaker 3: Yes, we can see your screen.
• 40:47
  Speaker 2: So there's this.
• 40:48
  Speaker 2: On the left side there's this link that's called Socratic AI.
• 40:53
  Speaker 2: So if you click that link it brings up this page which kind of explains what this AI learning assistant is about and how to use it, and blah, blah, blah.
• 41:00
  Speaker 2: But by default it's usually off, and so if we enable it, you'll see this little, you know, on the right side you'll see this little thing pop up and the idea there it's like, you know, you can open this up and you know it's pretty much just a chat bot, right, it's no different than a chat GPT kind of an engine that you have, right, it's just it's completely integrated into the material.
• 41:23
  Speaker 2: And so I'm just going to arbitrarily pick something.
• 41:26
  Speaker 2: And so what we do for example, one of the big things that we try to do is to like reinforce concepts, right, you know, reflective learning is sort of like a critical thinking, a critical aspect, like you need to have students pause from reading stuff and reflect on the material, and the easiest way to do this is to kind of have some very simple questions.
• 41:41
  Speaker 2: So at the end of every major section, for instance, you know, there's this little section quiz, and so when you click the section quiz, it auto-generates a quiz for you.
• 41:51
  Speaker 2: Now, this quiz is completely auto-generated by the LLM, right, so it's not something that's manually coded, so your question might be slightly different from my question and so forth.
• 41:59
  Speaker 2: And then you know I'm just going to randomly pick answers.
• 42:02
  Speaker 2: I don't know what's the right answer, so, and you can see down here, you know I got two of them correct, and then it gives you explanations to why is this the correct answer, why is it this answer, for instance?
• 42:13
  Speaker 2: And then you know, here I picked this particular thing and it says it's the wrong answer.
• 42:17
  Speaker 2: So it kind of helps you understand what's going on.
• 42:19
  Speaker 2: Mostly, this is really kind of meant to kind of just stop you from just kind of, you know, pouring through material, but it's really to kind of help you have a breather and as you kind of do this, um, you know you can also kind of pick certain concepts.
• 42:32
  Speaker 2: For instance, I can say, okay, send this context to the ai and then explain that, like I'm a five-year, six-year-old, right, um, so you can see that it's using Legos to explain this.
• 42:45
  Speaker 2: Nothing too surprising.
• 42:46
  Speaker 2: It's basically a built-in agent that understands how to do this and then also suggests other things that you might wanna ask.
• 42:54
  Speaker 2: Now, how do you sort of personalize this whole thing?
• 42:55
  Speaker 2: There are a couple of settings.
• 42:56
  Speaker 2: For instance, you can say, oh, I'm a complete beginner, I'm an animator person, this advanced in the future you could imagine we'll have like some customization where you can actually give it a prompt.
• 43:05
  Speaker 2: But it has to be structured.
• 43:05
  Speaker 2: Obviously we're academics so we tend to think very carefully about how to put things in.
• 43:10
  Speaker 2: So you know, this is no whole notion of Bloom's taxonomy which sort of understands how to ask questions to really test material.
• 43:16
  Speaker 2: So you know, the agent behind the scenes kind of knows all these things, um, and allows, you know, automatically does that.
• 43:24
  Speaker 2: And you know we want to gamify things in order to kind of make it a little exciting for people.
• 43:28
  Speaker 2: You know, it's like my daughter, it's like it's a duolingo binging.
• 43:32
  Speaker 2: I don't know why, but she loves the streak.
• 43:34
  Speaker 2: She wakes up and the first thing is like she's got to do the deal duolingo because the afraid to get the streak.
• 43:38
  Speaker 2: I don't know where that obsession comes from, because I sure don't have that.
• 43:43
  Speaker 2: But yeah, so we have, like, you know, like a dashboard that kind of generates, you know, stats about how you're doing and whatnot, and then you can download the report.
• 43:50
  Speaker 2: Mostly it's download because, you know, for me as a professor, I need to know if students are reading the material and so forth, so they can download it and upload it and say, ok, I finished the quiz for this particular reading assignment.
• 44:00
  Speaker 2: Some of the things that we're thinking about is, like, you know, you can think like, with all this kind of knowledge in place, as students are kind of progressing, you can actually have the agent learn about what the students are learning and not learning well, and then reinforce those concepts or make suggestions on hey, you should go back and read this chapter and that chapter and this particular section because it has the full context and so forth.
• 44:24
  Speaker 2: So a lot of stuff, stuff behind the scenes, but we have only enabled fairly basic, uh, basic sort of functionality.
• 44:30
  Speaker 2: But good enough that you know it helps people.
• 44:33
  Speaker 3: So yeah, that's fantastic would have been such a game changer to have when I was in university, you know all right, yeah, I use it.
• 44:41
  Speaker 1: Yeah, yeah, magic actually.
• 44:44
  Speaker 1: You know what I also appreciate.
• 44:45
  Speaker 1: I thought the illustrations you guys have in the in the book are actually pretty cool too, the Dali based things, and you put the actual prompt for Dali in there to explain how you generated that image to.
• 44:58
  Speaker 1: That was kind of like feeling tired.
• 45:01
  Speaker 2: I'm like I need some is.
• 45:02
  Speaker 1: Is that your?
• 45:03
  Speaker 2: yeah no, I thought that was pretty cool but you can totally think, like you know, what I just showed you is like, the fascinating thing is that that's just a single modality, right, that's just literally just text.
• 45:15
  Speaker 2: Things in the future are, like you know, multi-modalities, like you know you're pulling in from.
• 45:18
  Speaker 2: You know different kinds of.
• 45:19
  Speaker 2: You know, okay, I don't understand this image.
• 45:21
  Speaker 2: And you can add on all those things.
• 45:22
  Speaker 2: There's a lot.
• 45:23
  Speaker 2: So this concept of generative learning that I have is I think it's going to be radically for once.
• 45:28
  Speaker 2: I truly believe we can get to personalized education.
• 45:31
  Speaker 2: I truly believe this.
• 45:32
  Speaker 2: We've always talked about this in the past, but it's never scalable.
• 45:36
  Speaker 2: It's never been scalable, but through this technology, I think it suddenly becomes very scalable through this technology, I think it suddenly becomes very scalable.
• 45:47
  Speaker 1: Well, and also this is another differentiation of moving to more digital based learning, going back to the fact people do like the paper book and all that, but I think we're going to see more and more of a gap between what's capable in an online learning environment versus the book that you put your backpack.
• 46:01
  Speaker 1: So yeah, it's just another example of quizzes in the agency.
• 46:07
  Speaker 2: Yeah, the nice thing about having an online textbook for me has also been I can actually engage multiple modalities.
• 46:13
  Speaker 2: There's a text, then obviously you have pictures, which has always been around, but then now I embed videos in which the students absolutely love these short videos, and I'm not trying to invent everything from scratch.
• 46:24
  Speaker 2: There's plenty of amazing people putting amazing material online, but the key thing I think that we're all going to face as a major challenge in the future is not whether we have access to content.
• 46:35
  Speaker 2: You can say, well, do I even need a textbook in the future Because I have ChatGPT or I have Gemini, do I need it?
• 46:42
  Speaker 2: I think it actually becomes even more important to have a textbook because you need to know how to think about it with respect to how the world's kind of falling, which, I think you know, no generative agent will ever be able to kind of rationalize.
• 46:54
  Speaker 2: That you know, it is true.
• 46:56
  Speaker 1: So yeah, and you also have some pretty cool chapters in there toward the end around responsible AI, sustainable.
• 47:04
  Speaker 1: You also have some pretty cool chapters in there toward the end around responsible AI, sustainable AI.
• 47:08
  Speaker 1: That was pretty cool to read, at least to kind of get everyone on the page kind of a me almost, of what things are, what those terms are that we hear about.
• 47:15
  Speaker 2: Yeah.
• 47:15
  Speaker 1: Think about them as they're learning materials.
• 47:18
  Speaker 2: Yeah, that comes a lot from like things that I was taught, you know, in working with hyperscalers and so forth, the things that I've learned from the way they approach.
• 47:26
  Speaker 2: It is the feedback that they'd often give us.
• 47:28
  Speaker 2: Or we talk about these responsible ethics and sustainability and all these things as an afterthought, but in reality, it should actually be thought as part of designing a system.
• 47:38
  Speaker 2: It is.
• 47:38
  Speaker 2: It's not a separate team's kind of job, it's actually, you know, part of the core job.
• 47:43
  Speaker 2: In fact, just today, just before I came into the meeting, I was in our data science committee meeting and that's one of the things we were talking about.
• 47:51
  Speaker 2: Okay, where do we put critical thinking?
• 47:52
  Speaker 2: Do we put that as a separate class, or do we actually embed this directly into our core classes and have a week or two?
• 47:58
  Speaker 2: So every class is getting on the points over and over?
• 48:02
  Speaker 2: So then naturally, when you learn the material, you implicitly think about these things.
• 48:05
  Speaker 2: So, and then we're going to see money, yeah, things like the auditability.
• 48:12
  Speaker 1: Those things need to be.
• 48:13
  Speaker 1: Let's see, let's see, let's see Other questions here.
• 48:35
  Speaker 3: There was one from Amanda.
• 48:36
  Speaker 3: Yeah, is tinyMLOps already included in the course?
• 48:43
  Speaker 3: And I did see that there is ML Ops, but if you want to speak about that a little bit more, yeah, ml Ops is definitely in the course because the course is really structured into a key set of pillars.
• 48:55
  Speaker 2: One is focusing a bit on data, then talking about the training, then talking about the deployment and then maintenance.
• 49:01
  Speaker 2: It's like any engineering project you can build a bridge, but if you don't maintain the bridge you've got a serious problem right.
• 49:06
  Speaker 2: And so maintenance has kind of talked about it and TinyML Ops like there was a section where we kind of, like you know, try to emphasize how do things kind of change when you're talking about embedded ML Ops, you know.
• 49:17
  Speaker 2: So I kind of talk about things generally and then talk about okay, what are the unique challenges that start showing up when you're talking about tiny ml ops?
• 49:25
  Speaker 2: Because jenny kind of alluded to this at the beginning it is indeed very hard with the fact that you don't tend to have network connectivity or the fact that you have to estimate 50 of your power budget needs to be dedicated for comms, for barely sending any information up.
• 49:37
  Speaker 2: Like those are fundamental challenges that you don't you never, ever sweat in a traditional ml pipeline.
• 49:43
  Speaker 2: If you're on a kubernetes cluster, I doubt you're going to be sweating sending out a couple of network packets about, like you know how many predictions the model has made and you know whether it's not right and what it thinks about.
• 49:52
  Speaker 2: It's just noise in the massive collection, but in tiny amount it adds up like crazy, especially if you're doing laura van and so forth.
• 50:00
  Speaker 2: So, yeah, there is a section that kind of like talks about some of those unique challenges.
• 50:03
  Speaker 2: It's a good question.
• 50:11
  Speaker 3: This is a bit of a long one.
• 50:15
  Speaker 3: Good question, but it did get cut off.
• 50:17
  Speaker 3: So if they want to finish that question in another method, that would be great.
• 50:22
  Speaker 3: But essentially the question is thank you so much for writing this book.
• 50:27
  Speaker 3: They've already adopted the book's material for their EJI course at the Indian Institute of Science, which is amazing.
• 50:35
  Speaker 3: Their question is about the programmability of microcontrollers for EJI.
• 50:41
  Speaker 3: While C is preferred, the learning curve is steep and many beginning-level students find it difficult to understand its low-level interfaces.
• 50:50
  Speaker 3: I'm assuming the rest of the question goes on.
• 50:53
  Speaker 3: You know, like, how do you teach around this?
• 50:57
  Speaker 3: And you know, how do you adapt this for different types of deployments, for different microcontrollers that don't use C versus C++, etc.
• 51:03
  Speaker 3: Etc.
• 51:05
  Speaker 2: Yeah, I mean to be honest.
• 51:06
  Speaker 2: This is exactly why, when Marcelo kind of put the labs together and we discussed what level we want to share at a massive level, at a massive scale, the answer was picking up something like MATLAB right and Edge.
• 51:19
  Speaker 2: Impulse is practically the MATLAB for a lot of this.
• 51:22
  Speaker 2: At least that's the platform that I'm most familiar with.
• 51:29
  Speaker 2: Again, I'm sure there are other platforms, but we ended up choosing it because it was just easy and like a lot of students would just be able to easily onboard and the docs are pretty good for the students.
• 51:34
  Speaker 2: So I think like that was one way we kind of chose to kind of, you know, sidestep the fundamental issue and my hope ideally, ideally as educators, what we would do is we would take that as something students can be pointed to and be able to use the GUI.
• 51:51
  Speaker 2: I also do that in my own class, but I only do that for the first assignment, to get the micromicrophone working and then have them do a full deployment.
• 51:58
  Speaker 2: After that we rip the band-aid off and say, wow, that tool did a lot of magic.
• 52:03
  Speaker 2: Now let's actually understand all the magic pieces, because at the end of the day it's doing some pretty heavy lifting.
• 52:08
  Speaker 2: Yeah, we just slapped on a very nice GUI and understood how to work with users, not the engineers who actually have to build the stuff inside it, and so that's something that I think the students going to have to that's, on the instructors to be able to do, and so there's no easy answer down there, right?
• 52:28
  Speaker 2: I think this is also one of those areas that makes it difficult, because AI and so forth is really a confluence of embedded systems, however big or small the embedded system is.
• 52:38
  Speaker 2: So you need that traditional hands-on kind of an aspect, and then you have the ML component, and then, as Zaid mentioned earlier, it's the whole domain expertise kind of comes in.
• 52:48
  Speaker 2: So I think that's one of the things that makes us all unique is that an ML engineer cannot simply come in and be like oh you all do an NJI, I can do this.
• 52:57
  Speaker 2: It doesn't quite work that way, because suddenly they're hit with all these crazy constraints and all these quirky things, like you're pointing out about low level stuff, and I think that's what makes the ML systems engineers a unique breed of people who can do all this deployment.
• 53:15
  Speaker 3: And I mean I'm not going to lie starting at C code for a beginner is not easy.
• 53:19
  Speaker 3: But I will say, if the students work hard at it it will unlock every other code language for them, because if you know the fundamentals of C you'll know the fundamentals of almost every other language and Python, et cetera, et cetera.
• 53:33
  Speaker 3: So it's definitely a difficult thing for beginners, but I think it just takes a motivated student as well.
• 53:40
  Speaker 2: I think there are also, like these high-level tools that are kind of coming out right, like Micro, python and so forth.
• 53:44
  Speaker 2: I think it's just that now that we know that, okay, ubiquitous computing is likely on the horizon and it's possible, we're already deploying it.
• 53:52
  Speaker 2: I think it's bound to be that the languages are naturally going to evolve right, like Rust, and all these languages are also going to start kind of like transitioning over, because now there's a value proposition and there's a market for it that will actually pay the.
• 54:03
  Speaker 3: You know, you can hire people yeah, and that actually just uh bleeds into his next question.
• 54:10
  Speaker 3: That got cut off.
• 54:10
  Speaker 3: Or what are your thoughts on using micro python and its future for developing aji applications?
• 54:17
  Speaker 2: no, yeah, I, yeah, it just won't be my first go-to kind of a thing, because, yeah, it just won't be my first go-to kind of a thing, because I guess, you know, I'm like Jenny, you know and so forth we just kind of rip the band-aid off and say, sink in kids, like this is the real stuff, this is what makes us engineers and so, yeah, so I tend to be on that, you know, rusty side of like, well, it's just, you want an engineering title, but that's what makes us special.
• 54:45
  Speaker 2: It's like we can take the beating.
• 54:48
  Speaker 3: Yeah for sure, it's the longevity of really just.
• 54:53
  Speaker 3: It's so difficult but it's.
• 54:55
  Speaker 3: It's the success at the end that gets you through.
• 54:57
  Speaker 3: You know so.
• 54:58
  Speaker 3: But MicroPython I find is a really interesting.
• 55:00
  Speaker 3: We hear about it all the time at Agile Pools, like do?
• 55:07
  Speaker 3: I thought I find is a really interesting.
• 55:07
  Speaker 3: We hear about it all the time at a gym pools, like do you deploy with micropython, can you use micropython and um?
• 55:09
  Speaker 3: I think it's definitely up and coming, um, but I don't know if it's going to be used in you know a lot of industry deployments.
• 55:13
  Speaker 3: Because I think in industry deployments you really need like the bare, the bare bones code, cause you're not going to want it, you only have limited storage, space and memory and all those different types of constraints.
• 55:23
  Speaker 3: So at that point you usually need to do like the C level, c++ level code.
• 55:28
  Speaker 3: So I don't know.
• 55:29
  Speaker 3: I mean, we'll see how MicroPython evolves, but I'm curious as well Maybe your course will revolutionize that space.
• 55:36
  Speaker 1: Yeah, yeah, definitely.
• 55:41
  Speaker 1: Hey, we have a few minutes left.
• 55:44
  Speaker 1: I was going to say we should probably mention that there's a scholarship angle here, that you're with your GitHub stars.
• 55:57
  Speaker 2: Yeah, I was actually.
• 55:58
  Speaker 1: Do you want to explain that a little bit?
• 56:00
  Speaker 2: Yeah, I'll explain that.
• 56:01
  Speaker 2: Let me just share my screen because, yeah, I'm super grateful to the people who support these things.
• 56:07
  Speaker 2: So, back on our main page.
• 56:10
  Speaker 2: So folks who know me know that I'm a very big fan of outreach and supporting it.
• 56:17
  Speaker 2: I often say that if you hold Google's and Meta's and these hyperscalers responsible for all sorts of doing technology, incalers responsible for, you know, all sorts of doing technology in a socially responsible way, then I'd say, like, for education, it falls on academic institutions and nonprofits to kind of do the right thing, which is kind of help support education.
• 56:32
  Speaker 2: You know, outreach and support, so one of the things that the Edge AI Foundation coincidentally, this is not like it's been arranged in this way at all, coincidentally, it happened, like you know, the Edge AI Foundation, thanks to the feet here, um, you know, they're actually supporting these scholarship funds and the idea is that what we wanted to do is basically, you know, support two or three students.
• 56:53
  Speaker 2: It's a fair chunk of change for, like you know, having them actually work on problems that are relevant for the edgier foundation.
• 56:58
  Speaker 2: So, so every star, for instance, that we accumulate, you know, leads to some sort of support in terms of funding behind the scenes from the Edge AI Foundation.
• 57:05
  Speaker 2: Now, just to give a shout out in this, like you know, arduino and Seed actually did this last year where they were sorry.
• 57:12
  Speaker 2: This year they were donating these hardware kits that you know added up into several thousands of dollars, where we ran tiny workshops in development countries with their equipment, workshops in development countries with their equipment, and so now we're trying to progress through that, like from hardware kits to actually okay, we want to bring students on board who actually be able to support them financially, like actually pay for the housing and all that kind of stuff.
• 57:33
  Speaker 2: And it's a decent amount of money and for every star we get, like you know, adds up in terms of the expense that we can actually expense for them to be able to work on projects that are relevant to the industry.
• 57:45
  Speaker 2: So does that cover that, pete?
• 57:47
  Speaker 2: If not, please feel free to jump in.
• 57:48
  Speaker 2: Yeah.
• 57:49
  Speaker 1: No, I think that's great.
• 57:50
  Speaker 1: And, yeah, we have a scholarship fund.
• 57:53
  Speaker 1: I put the URL there If you want to take a look at it.
• 57:55
  Speaker 1: We're gonna be launching some fellowships in 2025 and some travel grants and underwriting workshops.
• 58:02
  Speaker 1: So, yeah, this is a great way to give back a little bit to help cultivate the next generation of AI leadership.
• 58:09
  Speaker 1: So appreciate your support, vijay, cool, well, we are a minute or so away from cutoff on our live stream here, so any last thoughts, vijay, you should leave us with other than go to mlsysbookai and start digging in.
• 58:27
  Speaker 2: Again.
• 58:27
  Speaker 2: My exciting thing, I would say, is that everybody wants to be an astronaut, nobody wants to be the rocket scientist.
• 58:32
  Speaker 2: So here's your opportunity and I guarantee you from a long-term perspective, I think there are going to be plenty of ML, quote-unquote model developers because the model development pipeline is becoming so easy, but there are going to be very few people, relatively speaking, can actually build those systems, and that's what it's going to come down to is being able to deploy these things, because the pipeline up in the front is maturing to the point it'll flow.
• 58:56
  Speaker 2: So you should think about that.
• 58:59
  Speaker 3: We're already seeing that, so you make a great point Sounds good.
• 59:01
  Speaker 3: Yeah.
• 59:01
  Speaker 3: But we're already seeing that, so you make a great point.
• 59:04
  Speaker 3: Yeah, but thank you and also a reminder it's open source, so feel free to contribute to the book if you wish, before it gets turned into a hard copy, and thank you so much for joining us today and everyone in the audience who asked such great questions.
• 59:17
  Speaker 2: Yeah, Thank you folks.
• 59:19
  Speaker 2: Thank you Pete and Jenny, Appreciate it you.