Latest / EDGE AI POD / How AI Zip Is Shrinking Models for a Device-First Future

Transcript

How AI Zip Is Shrinking Models for a Device-First Future · EDGE AI POD

• 0:09
  Speaker 2: So, Yubei, nice to meet you and I know we've met before too and AI Zip is actually a really interesting part of our community and it's good to be able to chat with you today, Nice to meet you too.
• 0:24
  Speaker 1: It's a pleasure to be here.
• 0:26
  Speaker 2: Well, I mean, why don't we kind of get into it?
• 0:28
  Speaker 2: I think you know, as I mentioned in the preamble before we started taping, there's lots of interesting companies in our community and doing all sorts of things.
• 0:40
  Speaker 2: You know there's like the whole I talk about the whole tech stack.
• 0:43
  Speaker 2: You know, from metal to cloud.
• 0:44
  Speaker 2: There's all these layers in there of companies.
• 0:48
  Speaker 2: It's like a team sport where you have to kind of work together to kind of actually solve problems with edge AI and tiny ML and stuff.
• 0:53
  Speaker 2: So, um, and you guys are really you're right in there, right in the stack, um, and so maybe you can kind of give us a little background on who who is.
• 1:03
  Speaker 2: Kind of what's the origin story?
• 1:04
  Speaker 2: Where did you guys come up with this idea?
• 1:07
  Speaker 2: What's the whole deal?
• 1:13
  Speaker 1: Definitely so.
• 1:14
  Speaker 1: Ai Zip essentially is a company that we specialize building the world's smallest and most efficient AI softwares, ai models that can, with the hope to essentially go into anything any devices, right so we're software only.
• 1:29
  Speaker 1: We'd like to partner with all the hardware leaders in order to bring AI to the next gen devices.
• 1:37
  Speaker 1: So over the years, we have partners with many very well known.
• 1:44
  Speaker 1: Over the years, we have partnered with many very well-known brands in this community.
• 1:50
  Speaker 1: So we won this year's CES Innovation Award together with SoftBank, and we won the Best Officer Award with Bosch, and we won the Best AI Product of the Year Award together with Analog Devices the best AI product of the year award together with analog devices, yeah, and so along the way, we have, uh, formed a lot of these.
• 2:10
  Speaker 1: Um, collaboration.
• 2:11
  Speaker 1: It has a team is almost a born out of an accident, right?
• 2:18
  Speaker 1: So we didn't have no plan to start a company at the beginning, right?
• 2:22
  Speaker 1: So then then COVID came right.
• 2:24
  Speaker 1: So COVID came.
• 2:25
  Speaker 1: Everyone was quarantined at home.
• 2:26
  Speaker 1: My friend came to me, asked me how do you actually build a working AI system?
• 2:33
  Speaker 1: I had no idea, and that was a shocking moment, because I've been always doing fundamental AI research for long and through hard learning, right?
• 2:41
  Speaker 1: So when we say fundamental AI research, we really somewhat times mean useless, right?
• 2:46
  Speaker 1: I know how to push the benchmark, I know how to write sometimes very good paper, but how to build a thing that actually work is the e-cheap part as an engineer, right?
• 3:00
  Speaker 1: So I don't really know.
• 3:02
  Speaker 1: So that was an interesting question.
• 3:05
  Speaker 1: And we start to experiment something and that eventually turned into a company because we all loved this and then seems like a very useful thing to do at the time.
• 3:18
  Speaker 1: Then we decided to create this company called AIZip in the end, late, late 2020.
• 3:25
  Speaker 2: I have to ask create this company called as, in the end, late, late, 2020.
• 3:27
  Speaker 2: I have to ask so the name.
• 3:28
  Speaker 2: You know, when I think of AI zip, I think of wind zip.
• 3:30
  Speaker 2: You zip your files together.
• 3:34
  Speaker 2: Was there some sort of inspiration there?
• 3:38
  Speaker 2: Was that the?
• 3:40
  Speaker 1: Yeah, so.
• 3:40
  Speaker 1: So the name the name was, you know, at the intuition of the name of the company is really the format that you compress your file, right?
• 3:51
  Speaker 1: So you want to compress AI into.
• 3:53
  Speaker 1: Essentially.
• 3:54
  Speaker 1: That's why we put AI and zip together, right?
• 3:57
  Speaker 1: So make this name and yeah, yeah, almost like you have a zip that AI is inside, right, so that's the intuition, yeah.
• 4:08
  Speaker 2: Yeah, you guys had actually a really impressive booth at CES.
• 4:12
  Speaker 2: I kind of stumbled upon it.
• 4:13
  Speaker 2: I was wandering around and all of a sudden I saw this huge thing and it was like, and there's all these people in there?
• 4:19
  Speaker 2: And I was like, oh AI, zip, wow, you guys are really kicking butt.
• 4:23
  Speaker 2: So yeah, you guys won an award with SoftBank at CES.
• 4:27
  Speaker 2: I think we actually just did a live stream about that.
• 4:31
  Speaker 2: That was for the aquaculture kind of fish farming solution.
• 4:36
  Speaker 1: Yeah, exactly so.
• 4:37
  Speaker 1: We have a very good collaboration with SoftBank and this year, that CES Innovation Award about, uh, smart farming for the next generation of smart farming, right, which is really, really important because, um, uh, you know, if you know about this market, um, a lot of the fish farm farmers actually went out of business because, during the well, everyone loves, loves sushi, right so, but we, we didn't know that a lot of these, uh, fish farmers actually went out of the business because a lot the the fish food price actually, uh, you know, increase.
• 5:16
  Speaker 1: The price increased actually is three, three times or so, right so, during the covet period, which is crazy, right so, now accounts for 70 of the cost of the fish farmers.
• 5:27
  Speaker 1: So, so, you know, if you're not careful and you can easily become a not a profitable business by doing that, right, because the food is very expensive and then we feed your the fishers right, so it's not going to.
• 5:40
  Speaker 2: You know a lot of these food are yeah, it's a, it's not going to you know a lot of these food are.
• 5:43
  Speaker 1: Yeah, it's a wasted right.
• 5:45
  Speaker 1: So the idea is really put AI there under the water to really gauge, count the fish and also gauge where the appetite is right.
• 5:55
  Speaker 1: So when to feed and how to feed, precisely almost like you try to water your plants, very precisely right To save water, and the same idea is here.
• 6:04
  Speaker 1: And that turned out to be a great success and now that evolves into the next generation of fish farmer infrastructure technology right.
• 6:17
  Speaker 1: So with these computer vision, power techniques and, of course, the Japanese artists, they are very creative.
• 6:23
  Speaker 1: They designed that booth with a lot of these fish almost like a movie.
• 6:28
  Speaker 2: that actually attracted a lot of the traffic yeah, so that we all loved that design, yeah yeah, no, it was really interesting yeah, yeah, that.
• 6:40
  Speaker 1: And we, we don't only stop there, right.
• 6:42
  Speaker 1: So now our collaboration extended to beyond fish, the farm, but in general, to agriculture, more categories, but also small language models, ultra efficient language model that can run on your device and do qna, do basic lag retrieval, argument generation, right, so that that is, uh, another large scope to for enterprise.
• 7:06
  Speaker 1: On device language model, we all know that language model is very has a huge market at the moment.
• 7:13
  Speaker 1: Right so so.
• 7:14
  Speaker 1: But on device market has been, you know, really underestimated, and we see that markets coming very quickly, and so we formed a team to tackle that.
• 7:28
  Speaker 2: So the models you were using for the fish farming, the aquaculture, those were like DNNs running on phones, basically, is that kind of?
• 7:39
  Speaker 1: Yes, it's vision models running on all level devices.
• 7:42
  Speaker 1: Move the.
• 7:44
  Speaker 2: DNNs maybe into the cameras themselves at some point and kind of do that Right.
• 7:49
  Speaker 2: Yeah, but now you're talking about also because I think you guys also did a talk at Embedded World in Austin in the fall around language model, running language models.
• 7:58
  Speaker 2: I think it was like on Brownfield systems or something you were demonstrating like how efficient you can get language models to run on fairly.
• 8:05
  Speaker 2: You know unaccelerated uh platforms right, and so is that a big Frontier for for right it was like how do you take language models and zip them up?
• 8:20
  Speaker 1: um, yeah, same philosophy.
• 8:21
  Speaker 1: So, um, language model is one of the new frontier that we start to work on.
• 8:28
  Speaker 1: Actually, we started to work on that more than two years ago, at the time when the CME published a paper called Tiny Story and then Sebastian Barbeck at Microsoft picked up that thread and started to lead the small language model charge at microsoft and start to talk about.
• 8:47
  Speaker 1: That was the early, some of the early work.
• 8:51
  Speaker 1: Um, tiny stories and uh, uh, textbooks, are you, are, you need, right?
• 8:57
  Speaker 1: So, this, all you need category papers, right, but we love that, right?
• 9:01
  Speaker 1: So, because we see that the future, a lot of the ai, the future of the AI, is going to be on device.
• 9:07
  Speaker 1: I can show you a few numbers to support that.
• 9:12
  Speaker 1: And, just a matter of fact, jason Huang, a media CEO, jason Huang just went to KnowPrior, gave a podcast and talked about what he, in his vision, uh, what would the future of the ai be?
• 9:27
  Speaker 1: And and he said that the future of the inference would be modhead scale.
• 9:31
  Speaker 1: You will have these large models and it will have these small models, right, so, just like a natural intelligence, you know, look at this world, right, so, we have human, but we also have a lot of dogs we have have cats, we have zombie spiders, we even have ants, right, so they're intelligent at every level.
• 9:46
  Speaker 1: So we see that the current inference, if you think about that, it's estimated right.
• 9:58
  Speaker 1: So 99% of the data is actually originated on device which is not surprising, because of you know where their sensors, their data the real world as we call it.
• 10:12
  Speaker 1: Exactly exactly.
• 10:13
  Speaker 1: But the surprising part another surprising part I think is more interesting than this is that if you think about the compute and actually add up the world's compute capacity in all these HPC centers right, it's roughly.
• 10:32
  Speaker 1: Also 1% compared to on-device, which is interesting because the sheer numbers of devices out there, right, so in your phone, everyone's pocket, in the cars, and smart cameras when you add this up, it's almost two orders magnitude more than in cloud computing resource.
• 10:52
  Speaker 1: But yet more than 95% of the compute, ai traffic, ai workload went to the cloud, which is interesting.
• 11:01
  Speaker 1: Majority of the investment also went to the cloud, which creates a huge opportunity for on device AI, right, so we see that opportunity coming very close and mainly because three market is three forces right, so the market is there for it and the hardware's capability is quickly increasing.
• 11:25
  Speaker 1: Just a matter of fact, if you think about seven years ago, apple's A11 chip, I believe that's the first generation that carries a neural engine, right, so the compute capacity for that was 0.6 T-ops and last year the compute capacity for that neural engine is 35, six fold more in seven years.
• 11:48
  Speaker 1: Then A19 is around the corner and in two years we're going to see.
• 11:54
  Speaker 1: And in two years we're going to see more than 100 T-ops compute capacity on device.
• 11:59
  Speaker 1: So a lot of these capacity is sitting in your pocket.
• 12:03
  Speaker 1: The language model will be on device.
• 12:05
  Speaker 1: A lot of these day-to-day elementary intelligence, everyday intelligence we call that, will be in your pocket, on your device, available there, right?
• 12:17
  Speaker 1: So that's the future we see.
• 12:18
  Speaker 1: So language model would not be an exception.
• 12:21
  Speaker 1: So we formed the group and start to tackle that problem, start to put AI agent on device.
• 12:28
  Speaker 2: Right, yeah, no, it makes total sense.
• 12:29
  Speaker 2: I mean, like you said, 90% of the data originates in the edge or the real world, and then they compute as you.
• 12:37
  Speaker 2: You it's an interesting stat to sort of add up all the compute out there, uh, relative to data centers, and you know, there's all those benefits of doing things on the edge, doing things where the data is created, so, uh.
• 12:48
  Speaker 2: So that's kind of your frontier, and so my understanding too, also, though, is like there's tops, but there's also like sort of memory constraints too, and one of the things that's sort of holding back language models and sort of resource constrained environments is, it is kind of a memory hog, um, so do you see like a lot of kind of progress happening in that space in the next couple years?
• 13:09
  Speaker 1: yeah, there are a couple very exciting opportunities in that direction.
• 13:14
  Speaker 1: So I would say the memory, yes, you're absolutely right, and we really start to realize that computer is not everything.
• 13:23
  Speaker 1: A lot of the times bottleneck can be your memory.
• 13:26
  Speaker 1: But one thing, you know, it's very interesting to see.
• 13:28
  Speaker 1: For example let's still take Apple as example A lot of the chip design is around how to resolve that memory bottleneck right, so that unified memory architecture right.
• 13:38
  Speaker 1: So all these designs are for the next generation of AI, Right?
• 13:42
  Speaker 1: If that's not designed for AI, a lot of these hardware design makes no sense, Right?
• 13:49
  Speaker 1: So I think the hardware leaders are preparing for that future and try to co design their architectures so that it will be appropriate for that future, and try to co-design their architectures so that it will be appropriate for the future AI workload.
• 14:01
  Speaker 1: And AI is just a new type of software, right, so you need the right hardware to support.
• 14:07
  Speaker 1: I think that this community is actually working on the right direction to form this co-design hardware and AI software co-design and that will be appropriate for the future AI workload on device yeah, yeah, no, for sure.
• 14:22
  Speaker 2: And uh, I mean, in the commercial space, as you know, there's um, there's always a challenge in terms of doing greenfield deployments versus you know more brownfield.
• 14:31
  Speaker 2: And how do you add, you know AI capabilities into existing deployments and things like that.
• 14:36
  Speaker 2: So I think that'll be interesting to see how that plays out.
• 14:38
  Speaker 2: But for sure, I mean, it's almost hard to believe, even from a few years ago, how much more compute is possible, and I mean you guys are focused on the software part to make it happen.
• 14:50
  Speaker 2: So that's pretty exciting.
• 14:51
  Speaker 2: Are you primarily focused?
• 14:53
  Speaker 2: Well, you talk about SoftBank.
• 14:54
  Speaker 2: So is Japan like a market for you?
• 14:56
  Speaker 2: Big market?
• 14:57
  Speaker 2: You're in California.
• 14:59
  Speaker 2: I mean, how big is AI Zip?
• 15:01
  Speaker 2: Are you a multinational conglomerate or what's going?
• 15:04
  Speaker 1: on.
• 15:04
  Speaker 1: Well, we are still pretty lean.
• 15:06
  Speaker 1: We have 30, some full-time and then including part-time, we're looking stop 50.
• 15:16
  Speaker 1: Uh, so majority of us are in the in california and, but we do have an employee from japan, from korea, from germany, from china, from taiwan, you know where we have the decentralized effect, right so.
• 15:30
  Speaker 1: But the team has been a very have a strong bonding, we have zero attrition, so, which is hard to do in the space, right.
• 15:38
  Speaker 1: So it's as such, a side, yeah and and um, yeah, so we love, we love japanese market and we myself, I'm a, I'm a judoka, I, I have done judo for eight years and, uh, you, you know, we love Japanese market and we have a lot of very fruitful collaboration with Japanese companies.
• 16:01
  Speaker 1: For example, softbank is one, and now we have a long term collaboration with Renesas and, and you know, we have a very strong and strategic partner to be released soon that's also in Japan.
• 16:21
  Speaker 1: So Japanese companies are leaders in many dimensions.
• 16:26
  Speaker 1: So, that's a very important marketplace.
• 16:29
  Speaker 2: Yeah, and they've been deploying technology leading technology in retail and healthcare and industry and robotics there for decades, so good place to be.
• 16:38
  Speaker 2: What do you think I'm?
• 16:41
  Speaker 1: surprised at how many people speak Japanese in ASM.
• 16:45
  Speaker 1: I was surprised, right, how many people at ASM actually speaks very fluent Japanese.
• 16:51
  Speaker 1: I was like, hey, wow, good for you guys.
• 16:55
  Speaker 2: That's a good skill.
• 16:55
  Speaker 2: What do you think about like model portability?
• 16:59
  Speaker 2: I mean, you're in the model business, you're in the software business.
• 17:01
  Speaker 2: The edge is notoriously fragmented and heterogeneous, right, you mentioned, there's always a new chip and a new thing, and they tend to have their own interesting, unique designs.
• 17:15
  Speaker 2: How do you manage portability of your models and software across these different, heterogeneous platforms?
• 17:22
  Speaker 1: I love this question, so.
• 17:23
  Speaker 1: So this question is actually very important.
• 17:25
  Speaker 1: I think, with the statement I'm going to make it, it's going to be very important for a lot of these hardware leaders to one thing we start to observe is that, of course, you need a lot of these deep hardware knowledge in order to make this happen.
• 17:40
  Speaker 1: And even though we are software only, we are very, very strong in terms of hardware expertise, and that definitely helps.
• 17:52
  Speaker 1: We built a lot of these AI what we call AI design automation tool chains that will deploy your model onto different target platforms.
• 18:02
  Speaker 1: So, just to name a few, there are Arm Arm is a good platform and then Rixofive, that's another one, and Qualcomm's Hexagon, and there are Cadence, hifive series.
• 18:16
  Speaker 1: So there are many, uh very important platforms, right.
• 18:19
  Speaker 1: So we formed automatically deployment uh flow for that and uh.
• 18:25
  Speaker 1: And one thing I do realize that that's a remain remaining challenge uh that I I think it need uh for the collaboration between us and the hardware leaders is really to form this new partnership to co-design hardware and software.
• 18:41
  Speaker 1: That's a gap we start to see.
• 18:44
  Speaker 1: I think NVIDIA is doing great in that dimension, right?
• 18:48
  Speaker 1: So, because we all know that Moore's Law has been slowing down, you know, just in terms of the hardware side.
• 18:57
  Speaker 1: So the new innovation a lot of the times come from co-design.
• 19:01
  Speaker 1: So, for example, nvidia designed their data center, so almost like a new compute unit, and then a lot of the co-design is there to facilitate communication here.
• 19:13
  Speaker 1: Try to compress a lot of the things into a smartphone factor, and Apple is also doing great right.
• 19:20
  Speaker 1: So there's an internal AI application development in Apple and there's strong hardware component there too, and then you can do this co-design right.
• 19:29
  Speaker 1: What do we see you know in for the rest of the community is that we really need to form this strong partnership so that the AI experts, software experts and hardware experts sit together and define the next generation of hardware.
• 19:45
  Speaker 1: By doing that, I think we can continue the scaling to improve the capability, rather than we think that hardware is going to be like this and we developed our network and then we realized, ah well, it's not deployable on this hardware, so that would be a pity.
• 20:03
  Speaker 1: So I think more seamless collaboration and integration would be definitely very important for this community Interesting.
• 20:12
  Speaker 2: Yeah, no, it's a tough nut to crack because I think a lot of the semiconductor partners want to kind of have their special sauce and their architectures that you know have their advantages.
• 20:22
  Speaker 2: But from a software side, sometimes that can be kind of challenging to sort of.
• 20:25
  Speaker 2: You know, anytime you start doing you know kind of abstractions and things like that, you get performance issues and size issues and things like that.
• 20:42
  Speaker 2: So, but it'll be interesting to see how that plays out.
• 20:45
  Speaker 2: I think that is something that I've heard a lot about.
• 20:47
  Speaker 2: So it's good that you guys are thinking about that.
• 20:49
  Speaker 1: So, yeah, yeah it's uh, just to compensate this point, I think it's's not merely a hardware issue.
• 20:57
  Speaker 1: So, for example, let's say you're selling Windows, so another hardware supplier, they build a new processor.
• 21:07
  Speaker 1: It would be a pity that a new processor cannot support Windows, because Windows is going to be a useful operating system to have on your hardware.
• 21:15
  Speaker 1: So to have these your hardware right, so to have these experts in the same room co-define the future I think that is important so that we don't waste our capabilities right.
• 21:27
  Speaker 2: True.
• 21:27
  Speaker 2: By the way, it's hard to get a new processor to support Windows just FYI, so I was in the Windows team for a little while.
• 21:33
  Speaker 2: It's a lot of work, yeah yeah, definitely, definitely.
• 21:36
  Speaker 1: Yeah, it's a lot of work.
• 21:37
  Speaker 1: Yeah, yeah.
• 21:38
  Speaker 2: Definitely, definitely.
• 21:38
  Speaker 2: Yeah, it's doable, but no, I hear you.
• 21:42
  Speaker 2: I think it's going to be an interesting bridge to cross.
• 21:45
  Speaker 2: I mean, I know that there's work at Microsoft, for example, with Onyx and some other frameworks and other things, so maybe there's some point there where there's more cohesion that we'll see Cool.
• 21:57
  Speaker 2: Any last words of wisdom about AI Zip that you want people to know about.
• 22:01
  Speaker 2: I think we got a pretty good view, but what else?
• 22:04
  Speaker 2: What's the thing that people should know about AI Zip that you don't think they know?
• 22:07
  Speaker 1: yet that there's a big opportunity in on-device resource, right?
• 22:21
  Speaker 1: So that has not been tapped into and the end game we see that is going to be the future of AI is going to be hierarchical and where you really have a lot of these gen models and all the models in between, they can do their useful things and have their specializations and meanwhile being ultra efficient and we envision really to be the portal of the future AI traffic, where there are sensors, there are AI models, and so that you know, even though very simple, but that could be the first station for a lot of this data to interact with the AI ecosystem, right?
• 22:59
  Speaker 1: So that's one of the vision we have.
• 23:01
  Speaker 1: Another thing that is quite unexpected, but over time I start to think more and more is the efficiency is actually more than just a application problem.
• 23:14
  Speaker 1: I shared with you several numbers at the beginning, very early on, but I want to share with you three, another three numbers, and the first one is always going to be the boring one, right?
• 23:26
  Speaker 1: So do any of you know how much power that the human brain consume?
• 23:32
  Speaker 2: It's a large model right.
• 23:33
  Speaker 2: So large I heard it's about 20 watts, 2025 watts.
• 23:37
  Speaker 1: Here we go.
• 23:38
  Speaker 1: Very nice, you are you, you are experts, right?
• 23:41
  Speaker 1: So it's about 20 Watts, right?
• 23:42
  Speaker 1: So that's about the power consumption of iPhone 15 max.
• 23:48
  Speaker 2: Oh, 15 max yeah.
• 23:51
  Speaker 1: Yeah, it's a.
• 23:52
  Speaker 1: It's a big device but, you know, still a lot more efficient than than the large language.
• 23:56
  Speaker 1: Unless you have a 600 billion parameter models and you deploy that onto two a 100 servers, You're looking at 10k watts, or four servers that would be 20k watts.
• 24:09
  Speaker 2: Oh yeah.
• 24:10
  Speaker 1: Huge.
• 24:10
  Speaker 1: That is the power gap, three hours magnitude, roughly.
• 24:14
  Speaker 1: But now I'm going to tell you the model difference.
• 24:18
  Speaker 1: We have this Tesla.
• 24:20
  Speaker 1: Tesla is very nice.
• 24:22
  Speaker 1: Everything is on device.
• 24:24
  Speaker 1: You have this on-device perception, 3d perception, planning, navigation in 2d space right, so, with a few billion parameters, that's the model size.
• 24:34
  Speaker 1: Right.
• 24:34
  Speaker 1: So, based on the foundation, then you build different heads for different detectors, right, right.
• 24:39
  Speaker 1: So that's the model.
• 24:40
  Speaker 1: But I will give you a example in the natural intelligence world, it's a jumping spider.
• 24:47
  Speaker 1: The jumping spider, right, also have eight cameras Two front, two on the side, two on the back, peripheral, and then another two, right, so eight cameras, eight cameras, 360 degree of vision right.
• 25:02
  Speaker 1: So build us, scan the environment and build a, a 3d perception and it can does not 2d navigation, but 3d navigation right, so it will look at its parade here.
• 25:13
  Speaker 1: Okay, I need to go down here and go through the bush and go up here and get to its prey, right, so that's very tricky.
• 25:21
  Speaker 1: Uh 3d navigation, right?
• 25:23
  Speaker 1: How many neurons does a jumping spider have to accomplish all of this?
• 25:29
  Speaker 1: I?
• 25:29
  Speaker 1: Don't know a thousand uh, no, not that small, but a few million neurons, right.
• 25:36
  Speaker 1: A few million neurons instead of a few billion neurons, right?
• 25:39
  Speaker 1: So that's against.
• 25:40
  Speaker 1: We're seeing this.
• 25:42
  Speaker 1: Three others magnitude difference and finally, I think that's going to be even more interesting in this this number is that you know we are talking about.
• 25:51
  Speaker 1: We love large language model, right.
• 25:52
  Speaker 1: So the scaling is so far the one of the most robust law we have found so far.
• 25:57
  Speaker 1: But the question is, uh, you know, every one and a half years we are seeing the training tokens become 10x more right, so we started Chinchilla, I believe it's 1.5 or so, and then now Lama3.
• 26:13
  Speaker 1: Last year was 15 trillions of tokens, right, so we're constantly seeing bigger, bigger.
• 26:18
  Speaker 1: Apple just released a new training set which is 200 trillion tokens.
• 26:23
  Speaker 1: That's a lot of tokens, but for humans, how many tokens can we actually acquire before the age of 20?
• 26:33
  Speaker 2: I don't know, I kind of blew the nerve and that number is not going to be a big one.
• 26:37
  Speaker 1: Oh well, let me do this simple exercise for you.
• 26:39
  Speaker 1: So let's say, start from day one.
• 26:41
  Speaker 1: We start to read text, right?
• 26:44
  Speaker 1: So we are going to read 10 words per second, which is a lot.
• 26:46
  Speaker 1: Usually people cannot read more than a couple words per second.
• 26:51
  Speaker 1: And let's assume each of the word is three tokens rather than 1.3.
• 26:55
  Speaker 1: 1.3 is the average token for general English text, I believe.
• 26:59
  Speaker 1: But let's say you're reading really complicated words, three tokens per word, and now that makes 30 tokens per second.
• 27:07
  Speaker 1: So you're going to do this 12 hours for 20 years, 12 hours per day for 20 years.
• 27:12
  Speaker 1: So that is going to be 9.6 billion tokens for 20 years.
• 27:19
  Speaker 1: That's three hours magnitude again.
• 27:24
  Speaker 2: Right, so you're seeing this.
• 27:26
  Speaker 1: Yeah, yeah, learning, learning, learning, learning, efficiency, model, efficiency, energy efficiency you constantly start to see this.
• 27:34
  Speaker 1: Three artists, magnitude difference, right?
• 27:37
  Speaker 1: So I think that's one of the next scientific frontier, right?
• 27:41
  Speaker 1: So how do we actually decipher this guy, right?
• 27:45
  Speaker 1: Just a matter of fact, right?
• 27:46
  Speaker 1: So when the steam engine actually first came out, right?
• 27:52
  Speaker 1: So so do you know what's the energy efficiency for a steam engine?
• 27:58
  Speaker 1: I?
• 27:59
  Speaker 2: don't't know, probably not much.
• 28:01
  Speaker 1: Not much.
• 28:01
  Speaker 1: It was a very tiny number.
• 28:04
  Speaker 1: I was surprised at how small that was.
• 28:06
  Speaker 1: I remember that was 0.2%.
• 28:10
  Speaker 1: Wow.
• 28:11
  Speaker 1: Right now, northeast, we constantly start to see 20% to 50%.
• 28:15
  Speaker 1: So it depends on what type of engine we're talking about, but that is the energy gap.
• 28:19
  Speaker 1: So energy efficiency gap improved by three hours magnitude also.
• 28:20
  Speaker 1: It depends on what type of energy we're talking about, but that is the energy gap, right.
• 28:21
  Speaker 1: So energy efficiency gap improved by three hours magnitude also right.
• 28:26
  Speaker 1: So I think data is a new type of entropy, right.
• 28:31
  Speaker 1: So, we're seeing this.
• 28:32
  Speaker 1: We're at the beginning of this revolution.
• 28:35
  Speaker 1: It's almost like we now have a steam engine which is not super efficient.
• 28:39
  Speaker 1: We now have a steam engine which is not super efficient, but I think getting this efficiency up would be an important factor is that towards egis actually, yeah, yeah, yeah, no, that's true.
• 28:52
  Speaker 1: Yeah, I mean unexpected thing yeah, and that's that's.
• 28:56
  Speaker 2: Uh, that's some fantastic analogies that you have there and I agree, I think we're sort of at the steam engine phase and, um, yeah, so we, we know that there's a lot of upside there, especially in efficiency.
• 29:05
  Speaker 2: We and I always say efficiency is the new currency.
• 29:08
  Speaker 2: You know, that's kind of the new thing is how efficient can you be with power and cost and other things to get these things done.
• 29:13
  Speaker 2: So cool.
• 29:15
  Speaker 2: No, this is really helpful to kind of get get a good picture of of where you guys are at and, yeah, looking forward to more success and I think for folks that don't know who AI Zip is, now you do.
• 29:26
  Speaker 2: So that's great and appreciate the time.
• 29:30
  Speaker 1: Awesome, all right, thank you.
• 29:31
  Speaker 1: Thank you for this interesting conversation.