Latest / Technology Now / Reducing barriers to entry to foundation model training

Transcript

Reducing barriers to entry to foundation model training · Technology Now

• 0:09
  Speaker 1: Hello, and welcome back to Technology Now. A weekly show from Hewlett Packard Enterprise, where we take what's happening in the world and explore how it's changing the way organizations are using technology. We are hosts, Michael Bird.
• 0:23
  Speaker 2: And Aubrey Lovell. And in the past couple of years, the world has seen the unprecedented growth and development of machine learning and AI applications along with a huge spike in demand for new systems. The continuous growth in demand has put a strain on the resources needed to successfully create these applications. This rising demand for AI services has, according to today's guest, stifled competition to supply AI to those with a lot of resources and cause uneven access for the rest of us. That's particularly true when it comes to accessing foundational models, the algorithmic building blocks of AI.
• 0:58
  Speaker 1: So in this episode, we are looking at reducing those barriers to entry and helping more people get access to foundation model training at less cost financially and to the planet. So if you are the kind of person who needs to know why what's going on in the world matters to your organization, this podcast is, of course, for you. Oh, and if you haven't yet done so, do make sure you subscribe on your podcast app of choice so you don't miss out. Right, Aubrey? Should we get into it?
• 1:28
  Speaker 2: Let's do it.
• 1:31
  Speaker 1: So you've mentioned the resource demand for AI on this podcast in the past, but according to data from research and markets, which we've linked to in the show notes the global demand for data center grade GPU's is growing at 32% per year hitting $15,400,000,000 in 2023 with the expectation it'll reach $61,700,000,000 wow, by 2028. And a lot of that is being provided by or at least used by just a few operators. Now according to findings from QY Research which we've linked to in the show notes. In 2023, the world's top 5 large language model or LLM developers acquired around 88.22% of the market revenue.
• 2:19
  Speaker 2: But giving smaller players even individual organizations the ability to create and train their own LLM could allow AI systems to become more efficient by streamlining them to one specific ideal use case, say materials research for example.
• 2:35
  Speaker 1: So how could we do it and just how much difference could democratizing AI make to our carbon emissions and our bottom lines? Well, Paolo Faraboszki is a fellow and VP at HPE currently leading the AI research lab at Hewlett Packard Labs. Paolo, welcome to the show. So what are the main barriers when it comes to allowing people to train their own foundation models?
• 3:04
  Speaker 3: Well, I think it's 3 main factors. Hardware, data, and skills. Right? So access to the AI hardware that's necessary to train some of the foundation models that are the state of the art is really become, something that only very few players can afford. While there are 100 of thousands of open source models out there that are freely available to the developers community, it's really, really hard and, you know, economically very impractical to get access to the compute power needs to train them.
• 3:38
  Data, as people say, is king in AI because most of the AI systems are trained on on data, and it's really hard to get good quality data. Most of the foundation models today are trained with datasets that are scrubbed from the public Internet, but the real gold mine is in the proprietary data that is hard to get, is dirty, it's, sparsely labeled, and there's a lot of work to be done there in making that data available to AI systems. Skills is the other, big gap. The competition to get access to both, AI developer skills and also system skills.
• 4:16
  Speaker 2: So there are currently only a few players with the ability and resources to build, train, and host their own models. What are the potential drawbacks to this, and how can democratizing this process help?
• 4:29
  Speaker 3: So right now, the dominant usage of AI, especially generative AI, is through the so called, API models, meaning models that are closed and proprietary only can be consumed through a cloud hosted interface. This is a challenge because, effectively, the players that control those proprietary models also dictate to the downstream, usage of all the application that can be built around those models. So it's also hard to know how they were trained. So it's kind of lack of transparency and lack of flexibility in how you deploy outside of the way in which the, closed models are intended by their developers is a big challenge. The second one is the fact that these model providers do get access to the data, for example, the prompts and the feedback for to the prompts that the users provide to a system.
• 5:22
  So this is, effectively feeding into the system, making the system better. So, they further gain advantages versus the competition by having access to the data. It's a little bit like the search market where effectively the more you search, the better the search engine becomes. And so it's again kind of growing the gap between the few players and the rest. And the third one is the investments.
• 5:45
  Right? The investments required to be a big player in this space are really getting out of control. So not many other place can afford the capital expenditure whether you buy or rent the hardware needed to build a state of the art model so that is also further complicating the issue. So, I mean, the benefits of democratizing or making it more broadly accessible would be to effectively counter these three disadvantages I just mentioned. If you can lower the cost, we can make it more broadly accessible.
• 6:17
  It will open up a lot of innovation and opportunities across the board. A little bit like what the PC market did to the computing market in the eighties. The open source movement in general made, a lot of the tools like compilers, operating system, and so on more accessible. Right now, in some of the AI markets, I can see a bit of a kind of a vertically integrated lock in, similar to what happened in the eighties with mainframes, including proprietary software. And, again, the open source.
• 6:48
  I think something similar is likely to happen. I think the open source community and the availability of open model that could be inspected, analyzed, you know, they can be tested, they can be improved by the community at large is going to be a very, very important component of all of this.
• 7:07
  Speaker 2: So we've talked a little bit about open source, looking at other factors. How can we reduce the barriers to entry for AI and foundation models?
• 7:15
  Speaker 3: It's gonna be a combination of factor. There's gonna be several two x gains coming from a variety of different dimensions. And altogether, hopefully, if they stack on top of each other, they can get into this 2 to 3 odds of magnitude improvement. We're already seeing some early signs of that. And there's a large amount of inefficiency in running that system if you don't know what you're doing.
• 7:39
  That, for example, the high performance and supercomputing community has developed and perfected over the last several decades. And some of these technology are becoming available in the eyespace, and companies that are strong in the supercomputing space are becoming key players in that space as well. So that's sort of, you know, the the low hanging fruit, and everybody is sort of going after that efficiency improvement right now. Then there is a set of techniques that are addressing the data side from not the easy, but the obvious, approach of saying, well, I can I train with less data? And that will definitely cause a reduction in the cost.
• 8:17
  There's also some kind of deeper technology work that is showing that perhaps you don't need as many bits as people were using for each of the parameters of a model. This would be groundbreaking because, effectively, you know, you can build multipliers in hardware that are a lot cheaper when you're dealing with well, you don't really need a multiplier when you're multiplying by 01 and minus 1. Right? So major improvement in hardware. There's there's a big push in the the economic community to to do so called mixture of experts, which are it's a sparse of the activated models.
• 8:48
  In other words, you don't need to treat the model as just one gigantic monolithic entity, but you can effectively target it, break it into different specialized models. And by breaking, like, big model into small models, you gain a lot of efficiency because, you know, the sum of the square of small models is a lot smaller than the square of the big model. Right? So that's where the training cost kind of go down. And the last thing I wanna mention is the fact that there are lots of new players coming up in the accelerator market.
• 9:21
  So far, the hardware market has been also dominated by a few players, but we're seeing as in any market some correction that hopefully is gonna pick up the competition. And we're tracking this space with a lot of attention to see some of the new, you know, hardware initiatives that are very promising these days.
• 9:42
  Speaker 2: Alright. Thanks, Paolo. It's a fascinating area, and we can't wait to hear more. And we'll be back with you in a moment, so don't go anywhere. Okay.
• 9:51
  Now it's time for Today I Learned, the part of the show where we take a look at something happening in the world we think you should know about. Michael, what do we got for today?
• 10:00
  Speaker 1: Well, Aubrey, it's a very, very, very exciting one. Researchers at a university in Israel have taught a patient to speak by imagining sounds and then having a computer translate it into speech. Yeah. Exactly. And it's the first time thought to speech has ever been tested.
• 10:21
  It's the first time I've ever said that phrase as well. So, essentially, what researchers did is they piggyback onto the treatment for a patient with a special kind of epilepsy, which causes seizures to start deep within the brain. To pinpoint the cause of the seizures ahead of surgery, the patient has electrodes implanted deep within the brain much deeper than scientists alone would be able to go to for research purposes. They then ask the patient to say a word and read their brain patterns. They then asked them to imagine the word and then compared the signals.
• 10:53
  Now using machine learning, they were able to compute what sounds the patient was thinking of and then translate that into speech. How awesome is that?
• 11:03
  Speaker 2: That's pretty awesome. Yeah. I would definitely want to know how to use that with my spouse and family.
• 11:09
  Speaker 1: Me too. Now so far the research only used 2 syllable sounds, but it's thought that with more research the system could be used to help paralyzed patients communicate and be used to create a full library of the individual brain communication patterns of patients with early stage ALS or motor neuro disease, allowing them to communicate even once they start to lose muscle control. Incredible stuff.
• 11:35
  Speaker 2: That really is a remarkable story and and very cool. Thanks, Michael. Alright. Now it's time to return to our guest, Paolo Faraboschi, to talk about reducing the barriers to entry for foundation models in AI. So, Paolo, you've written a paper on the topic, which we'll link to in the show notes.
• 11:56
  Could you tell us a little bit more about your research?
• 11:59
  Speaker 3: Yeah. This paper was a result of the development of what we called an analytical framework, which is effectively a way to reason about a problem quantitatively. And we did a lot of research as in collecting, information from, studies and trends and market analysis and technology analysis. And we tried to put it together to understand where the costs were going, what were the the potential remediation path. I mentioned some of the technologies right now.
• 12:31
  We also highlighted the fact that there is a second, what we call the revolutionary path, that takes a a more radical look at some of these problems. And, you know, it's a very interesting field of research. It kind of claims we're all doing it wrong and that we really should be looking at the way in which human brains, learn, and they can do very amazing things within, like, a 100 watts footprint. So there's value point there. So perhaps we do need to look at, you know, physics and neuromorphic, inspired models that don't use the brute force, both on the computational and the data side of the house and have the promise to do not just 3 orders of magnitude, but like 5 orders of magnitude of reduction in complexity and cost of of doing generative AI.
• 13:23
  Now this research is very promising, but of course, it will only bear fruit in perhaps a decade or or more. So we can't wait for it, so we need to, invest in it but at the same time we also need to invest in what we call the evolutionary roadmap which is the collection of technology that I described a little bit before.
• 13:42
  Speaker 1: So, Barlow, what are the next steps here for anyone who is looking to get into the space?
• 13:47
  Speaker 3: The burden is gonna, shift towards the end users. I mean, the organizations that are getting into AI and they're trying to figure out how to implement AI for their own business or processes. And I think making sure that they understand, you know, how to pick their partners carefully is important. Right? They need to very carefully evaluate who they're gonna take as partners, where it makes sense to play, where it doesn't make sense to play, and and sometimes, you know, the there's always a gut reaction of saying, oh, well, I'm gonna build my own model from scratch, and then, you know, 6 months later, you realize you don't have the skills you don't have.
• 14:27
  The compute power and the investments would, you know, break the bank. Right?
• 14:31
  Speaker 2: Okay. Final question. Why should our listeners be paying attention to the issue of barriers to entry and AI model training?
• 14:39
  Speaker 3: Well, AI is really gonna be embedded everywhere. These days, we're treating it as something special. Right? Even when we're looking at regulatory bodies and things like that, they wanna treat AI as some kind of special topic. Right?
• 14:53
  But if we go forward 10 years, we'll probably have a laugh at the way in which we were attempting to curb it and to reason about it. It's gonna be like energy and electricity. It's gonna be really embedded in everywhere. So so if we really believe that that AI is gonna be embedded everywhere, and I do, then it's particularly important to understand how to control the generation of AI, which is sort of where everything starts. And that's where many of these discussion we just talked about are covering.
• 15:25
  Speaker 2: Thanks so much, Paolo. It's been great to talk. And you can find more on the topics discussed in today's episode in our show notes.
• 15:35
  Speaker 1: Right. Well, we are getting towards the end of the show, which means it is time for this week in history. A look at monumental events in the world of business and technology, which has changed our lives. Now, Aubrey, what was the clue last week?
• 15:51
  Speaker 2: Just taking a stretch here and putting my glasses on. So the clue last week was it's 1774, and you'll be breathing easy after this discovery. Did you get it?
• 16:02
  Speaker 1: Yeah. I had no clue.
• 16:03
  Speaker 2: Okay. Me neither.
• 16:04
  Speaker 1: No clue. Not a clue.
• 16:05
  Speaker 2: Well, it definitely was a tricky one, but the answer is the discovery of oxygen. Pretty big deal.
• 16:11
  Speaker 1: That's a big one.
• 16:12
  Speaker 2: It is a big one. Right? This week, 250 years ago by priest, philosopher, and scientist Joseph Priestley. How convenient of that last name. Priestley was working out of a laboratory in Wiltshire in the UK when he made the discovery that by heating red mercuric oxide, he released a colorless gas that a candle would happily burn in and a mouse he used as a test subject could thrive in.
• 16:37
  Other scientists and friends of Priestley continued the research which quickly identified oxygen as a separate element, dominant part of the atmosphere and important part of combustion. Very big discovery. Priestley was a close friend of Benjamin Franklin and Thomas Jefferson and favored both the American and French revolutions. In 17/94, he fled to Pennsylvania where he established a new laboratory and one of the finest libraries of chemistry of the age, as well as making numerous other discoveries along the way before his death in 1804. Pretty cool dude.
• 17:11
  Speaker 1: Pretty cool dude. How do you know it was red mukuric oxide if oxygen hadn't been invented yet?
• 17:16
  Speaker 2: That is a question for producer Sam. I have no idea.
• 17:21
  Speaker 1: Maybe he didn't know it was called But
• 17:22
  Speaker 2: but a good good challenge. Good challenge Yeah. For sure.
• 17:25
  Speaker 1: Hang on a second. This timeline doesn't match up.
• 17:28
  Speaker 2: Wait a minute.
• 17:29
  Speaker 1: Wait a minute. Amazing. Amazing. Thank you, Aubrey. Now the clue for next week is, that was the drum roll.
• 17:38
  It's 1930, and this was one small birthday. One small birthday. Any ideas, Aubrey?
• 17:48
  Speaker 2: Nope. I don't have a clue.
• 17:49
  Speaker 1: Nope. Alright. Well, we'll find out next week. And that of course brings us to the end of technology now for this week. And a huge thank you to our guest Paolo Faraboski, fellow and VP at HPE, currently leading the AI research lab at Hewlett Packard Labs.
• 18:05
  And to you, thank you so much for joining us. Technology Now is hosted by Aubrey Lovell and myself, Michael Bird. And this episode was produced by Sam Datapolin with production support from Harry Morton, Zoe Anderson, Alicia Kempton, Alison Paisley, Alyssa Mitri, Camilla Patel, and Chloe Sewell.
• 18:23
  Speaker 2: Our fabulous social editorial team is Rebecca Wissinger, Judy Ann Goldman, Katie Guarino, and our social media designers are Alejandra Garcia, Carlos Alberto Suarez, and Ambar Maldonado.
• 18:35
  Speaker 1: Technology Now is a low street production for Hewlett Packard Enterprise, and we'll see you the same time, same place next week. Cheers.
• 18:42
  Speaker 2: Cheers.